Dynamic updating of a composite image

ABSTRACT

A smartphone may be freely moved in three dimensions as it captures a stream of images of an object. Multiple image frames may be captured in different orientations and distances from the object and combined into a composite image representing an image of the object. The image frames may be formed into the composite image based on representing features of each image frame as a set of points in a three dimensional point cloud. Inconsistencies between the image frames may be adjusted when projecting respective points in the point cloud into the composite image. Quality of the image frames may be improved by processing the image frames to correct errors. Further, operating conditions may be selected, automatically or based on instructions provided to a user, to reduce motion blur. Techniques, including relocalization such that, allow for user-selected regions of the composite image to be changed.

BACKGROUND

As mobile phones become more sophisticated, they incorporate componentsthat make these devices versatile and practically indispensable to theirowners. Most existing smartphones include a camera and various inertialsensors, such as an accelerometer and gyroscope. The smartphones canalso include a proximity sensor, magnetometer, and other types ofsensors that allow using the phones for a wide array of functions.

Smartphones can be used to capture information with their cameras. Usersvalue a smartphone's ability to take pictures since this feature allowseasily capturing memorable moments, documents, perform bank transactionsand a wide array of other possibilities. Images of simple scenes aregenerally acquired—a photograph or a video. Existing smartphones do nottypically analyze the acquired images, and the user has to visuallyexamine each image and decide whether it is of an acceptable quality.

Further, existing smartphones can be used to acquire good quality imagesof small documents, such as a business card or check for deposit in abank. However, to image a large object, a smartphone needs to be held ata distance from the object. As a result, an image of a poor quality andlow resolution is typically obtained, with details, such as text, beingblurred and not easily recognizable.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings are not intended to be drawn to scale. In thedrawings, each identical or nearly identical component that isillustrated in various figures is represented by a like numeral. Forpurposes of clarity, not every component may be labeled in everydrawing. In the drawings:

FIG. 1 is a sketch of an environment in which some embodiments of theinvention may be implemented;

FIG. 2 is a block diagram of components of a mobile device in which someembodiments of the invention may be implemented;

FIG. 3 is a schematic diagram of processing of image frames forming acomposite image captured as an object is being imaged, improving qualityof the composite image and providing feedback to a user, in accordancewith some embodiments;

FIG. 4 is a flowchart of processing of image frames to improve a qualityof a composite image and providing feedback to a user, in accordancewith some embodiments;

FIG. 5 is another flowchart of a process of processing of image framesto improve a quality of a composite image and providing feedback to auser, in accordance with some embodiments;

FIG. 6 is a flowchart of processing of image frames to improve a qualityof a composite image and controlling operation of a camera of asmartphone, in accordance with some embodiments;

FIG. 7 is a sketch of a representation of image frames in a threedimensional point cloud, in accordance with some embodiments;

FIG. 8 is a flowchart of a process of building a composite image byrepresenting features of image frames in the three dimensional pointcloud, in accordance with some embodiments;

FIG. 9 is a schematic diagram that illustrates adjusting a pose of animage frame by aligning the image frame with a preceding image frame, inaccordance with some embodiments of the invention;

FIGS. 10A, 10B, 10C and 10D are schematic diagrams illustrating anexemplary process of scanning a document by acquiring a stream ofimages, in accordance with some embodiments of the invention;

FIGS. 11A and 11B are schematic diagrams of an example of adjusting arelative position of an image frame of an object being scanned byaligning the image frame with a preceding image frame, in accordancewith some embodiments of the invention;

FIGS. 12A, 12B, 12C and 12D are schematic diagrams illustrating anexemplary process of capturing a stream of image frames during scanningof an object, in accordance with one embodiment of the invention;

FIGS. 13A, 13B, 13C and 13D are conceptual illustrations of a process ofbuilding a network of image frames as the stream of image frame shown inFIGS. 8A, 8B, 8C and 8D is captured, in accordance with someembodiments;

FIGS. 14A, 14B and 14C are schematic diagrams illustrating anotherexample of the process of capturing a stream of image frames duringscanning of an object, in accordance with some embodiments of theinvention;

FIG. 15 is a conceptual illustration of a process of building a networkof image frames as the stream of image frame shown in FIGS. 10A, 10B and10C is captured, in accordance with some embodiments of the invention;

FIG. 16 is a flowchart of a process of improving image quality by digitremoval, in accordance with some embodiments of the invention;

FIGS. 17A, 17B and 17C are schematic diagrams illustrating the processof FIG. 16 of improving image quality by digit removal, in accordancewith some embodiments of the invention;

FIG. 18A-18B are conceptual illustrations of K-frames and P-frames in astream of image frames, in accordance with some embodiments of theinvention;

FIG. 19 is a flowchart illustrating a process of image frame acquisitionfor real-time processing and display, in accordance with someembodiments of the invention;

FIG. 20 is a flowchart of updating a reference image, in accordance withsome embodiments of the invention;

FIG. 21 is a flowchart of a process of improving image quality byremoving a reflection, in accordance with some embodiments of theinvention;

FIGS. 22A-22D are schematic illustrations of an example of improvingimage quality by removing a reflection by using motions of a smartphone,in accordance with some embodiments of the invention;

FIG. 23 is a flowchart of a process of segmenting an image, inaccordance with some embodiments of the invention;

FIG. 24 is another flowchart of a process of segmenting an image, inaccordance with some embodiments of the invention;

FIG. 25 is a flowchart of a process of applying optical characterrecognition techniques to an image and assessing quality of theapplication of optical character recognition techniques, in accordancewith some embodiments of the invention;

FIGS. 26A-26D are conceptual illustrations of a process of segmentationof an image and application of optical character recognition techniquesto the image, in accordance with some embodiments of the invention;

FIG. 27 is a flowchart of a process of adaptive camera control to reducemotion blur when capturing a sequence of images, in accordance with someembodiments of the invention;

FIGS. 28A-28B are schematic illustrations of exemplary messagesindicating to a user information relating to operation of a portableelectronic device, in accordance with some embodiments of the invention;

FIG. 29 is a flowchart of a process of dynamic updating when capturing asequence of images to use in a composite image, in accordance with someembodiments of the invention;

FIGS. 30A-30F are illustrations of a user interface of an exemplaryportable electronic device at multiple stages in a process of capturinga composite image and then selecting regions to update in order toremove and/or add an object into the composite image, in accordance withsome embodiments of the invention; and

FIGS. 31A-E are illustrations of a user interface of an exemplaryportable electronic device at multiple stages in a process ofdynamically updating an image of a scene, shown side by side withcontemporaneous illustrations of the scene in reality, in accordancewith some embodiments of the invention.

DETAILED DESCRIPTION

The inventors have developed image processing techniques that enable asmartphone, or other portable electronic device, to capture images withhigh quality and/or of large objects. These techniques may be based onconstructing a composite image from multiple image frames of an object.The image frames may be combined such that the extent of the objectrepresented in the composite image exceeds the extent depicted in asingle image frame. Such an approach may enable imaging with thesmartphone camera held close to the object such that each image framerepresents only a portion of the object, but with higher resolution thanif the phone were held far enough from the object to capture the entireobject in a single frame. Alternatively or additionally, an image of anobject may be formed from multiple image frames by using some imageframes to improve the quality of the composite image.

Some of the techniques described herein are based on approaches forcombining image frames captured with a smartphone, or other portableelectronic device. The combination may extend the composite image beyonda single image frame or may replace a first segment of the compositeimage, derived from a first image frame or a first subset of imageframes, with a second segment, derived from a second image frame or asecond subset of image frames. The replacement may remove a poor qualitysegment of the image, thereby improving the overall quality of thecomposite image. Alternatively or additionally, the replacement may addor remove objects from an image of an overall scene. The portions of theimage that are replaced may be identified by user input or in any othersuitable way. Accordingly, the techniques described herein includetechniques for identifying segments of an image for replacement and/ortechniques for identifying other segments that may be suitablereplacements and/or techniques for forming a composite image from imageframes or image segments.

Such techniques for combining image frames may be based on identifyingimage features in image frames and, in combination with positional datafrom the smartphone, representing the features as a three dimensionalpoint cloud. Sets of points, each set representing features extractedfrom an image frame, may be positioned within a common frame ofreference representing the composite image. Initially, the sets may bepositioned within the point cloud based on position information of thesmartphone at the time the associated image frame was captured. Thispositional information may include information such as the direction inwhich the camera on the phone was facing, the distance between thecamera and the object being imaged, the focus and/or zoom of the cameraat the time each image frame was captured and/or other information thatmay be provided by sensors or other components on the smart phone.

As each set of points is added to the point cloud, its three-dimensionalposition may be adjusted to ensure consistency with sets of pointscontaining points representing an overlapping set of features. Theadjustment may be based on projecting points associated with multipleimage frames into a common plane of reference. When there is overlapbetween the portions of the object being imaged represented in differentimage frames, adjacent sets of points will likely include pointscorresponding to the same image features. By adjusting the threedimensional position associated with each set of points to achievecoincidence in the plane between points representing the same features,quality of the composite image can be improved. In this way, a coarsealignment of image frames, associated with the sets of points, may beachieved.

A finer alignment also may be achieved to further improve image quality.As more image frames are gathered and additional sets of points areadded to the point cloud, the relative position and orientation of thesets of points may be adjusted to reduce inconsistencies on a moreglobal scale. Such inconsistencies may result, for example, from errorsin inertial sensor outputs that accumulate as the smart phone is movedback and forth, nearer and further from an object being imaged.Inconsistencies may also result from an accumulation of small errors inalignment of one set of image points to the next as part of the coarsealignment.

Regardless of the number and nature of alignment processes, processingcircuitry may maintain an association between the points in the cloudand the image frames from which they were extracted. Once the relativeposition, orientation, zoom and/or other positional characteristics aredetermined with respect to a common frame of reference for the sets ofpoints, a more accurate mapping between the image frames and thecomposite image may be determined. The composite image then may berendered by combining separate image frames with this mapping.

Yet a further quality improvement might be achieved by selecting fromamong multiple image frames to provide detail of one or more segments ofthe composite image. Because the smart phone may be moved in multipledimensions, the same portion of an object may be imaged from multipleorientations or at multiple different times. As a result, differentimage frames may depict the same portion of the object with differentquality. The point cloud enables the image frames that depict the samesegment of the composite image to be identified. In some embodiments,techniques may be employed to identify relative quality levels of imageframes from which information about the same segment may be obtained.Using relative quality information, information from one or more ofmultiple image frames representing the same segment may be identifiedand used in rendering the composite image.

Any suitable technique may be used to determine relative image quality.In some embodiments, for example, when the object being imaged is adocument, optical character recognition techniques may be applied tosegments of the image to assess the quality of those segments.Alternatively or additionally, image processing techniques may beperformed to determine whether features within an image segmentconstitute reflections or shadows. Such techniques, for example, allowidentification of segments of low-quality to be replaced by other imagesegments depicting the same portions of the object being imaged withimage segments of a higher quality.

Moreover, in some embodiments, when none of the image framesrepresenting a segment has suitable quality, image fill techniques maybe used to avoid distracting features in the composite image. As aspecific example, a portable electronic device may be used to acquire animage of a piece of paper or other object held by a user. In thatscenario, the user's finger may appear in captured image frames.Processing may determine a segment of the composite image depicting theuser's finger. Further processing may replace that segment with a lessobjectionable segment, such as a segment of a background color of thedetected object.

Yet a further improvement in image quality may be achieved by processingportions of the composite image as it is being formed and using resultsof that processing to guide acquisition of image frames to complete thecomposite image. In some embodiments, image capture, processing anddisplay as described herein may be performed within a smart phone orother portable electronic device. Accordingly, techniques as describedherein to identify segments of the composite image of low quality may beexecuted in real-time—meaning that low-quality segments may beidentified while the user is moving a smart phone to acquire an image ofan object. This real-time identification of low-quality segments may beused to render a display indicating to the user areas of the object thatshould be imaged again to improve image quality.

As an example of feedback to a user based on real-time processing, insome embodiments, real-time processing of a composite image may identifyreflections or other image artifacts that are being captured. Inresponse, direction may be output to a user to alter the orientation ofthe smart phone to avoid reflections or other image artifacts. Asanother example, processing may detect that the object being scanned isa sheet of paper. The size of the paper may be determined automaticallysuch that the extent of the sheet of paper that has been imaged may becompared to the detected page size, allowing portions of the page thathave not been imaged to be identified. Feedback to the user may directthe user to image those portions of the page.

An enhancement on this technique, which may be used in some embodiments,entails identifying that the page is warped such that the detected pagesize is not an accurate representation of the extent of the object to beimaged. Detecting warpage may improve the accuracy of feedback providedto the user about portions of the object that need to be imaged.Detecting warpage alternatively or additionally may be used to applyde-warping algorithms to the captured image, further improving imagequality.

As yet a further technique that may be used to improve image quality,information obtained while processing image frames in real-time may beused to adjust capture conditions for subsequent image frames. In someembodiments, quality metrics computed on captured image frames may yieldan indication of the resolution required for acceptable image quality.Based on these quality metrics, the average resolution of capturedimages may be adjusted. In some embodiments, the stream of image framesmay be captured, and different image frames in the stream may havedifferent resolutions. The average resolution may be adjusted bychanging the ratio between the number of higher and lower resolutionimage frames in the stream. Alternatively or additionally, real-timeprocessing of image frames may be used to adjust other hardware settingscontrolling capture of subsequent image frames.

In some embodiments, the quality metrics may relate to blur. The blurmay be an indication of motion of a portable electronic device used tocapture image frames or may be an indication of motion of objects withina scene being imaged using the portable electronic device. Alternativelyor additionally, the quality metrics may relate to camera gain and/orcurrent lighting conditions.

Such quality metrics may be used to conditionally trigger correctiveaction. The corrective action may entail automatically adjusting captureconditions and/or may include providing a message to a user of theportable electronic device relating to operation of the device. Such amessage, for example, may signal to the user to turn on a flashassociated with a camera on the portable electronic device and/or toslow motion of the portable electronic device while scanning the scene.

In some embodiments, capture conditions may be adjusted by selecting anexposure time and/or camera gain that provide acceptable image qualitybased on sensed motion of the portable electronic device and/or objectswithin a scene being imaged. In some embodiments, exposure time and gainmay be set in an iterative process, with exposure time beingpreferentially changed. Such an approach may lead to operation at anacceptable level of blur with a low gain, which may reduce image noiseand increase image quality. Such camera parameters (e.g. exposure time,gain, and flash) may be controlled in real-time to reduce motion blur tobe at acceptable levels when capturing a series of images to form acomposite image.

At times, the user may desire to change one area of the composite imagewhile keeping the remaining composite image. Accordingly, in someembodiments, a portable electronic device may provide a user interfacethrough which a user may dynamically change a specified area after acomposite image is formed. Such a technique may enable the user toremove an undesired area in the image by being able to select a regionto recapture.

As an example, the user may mark an area which was captured incorrectlyand update the faulty area again without having to capture the entirecomposite image. In another example, the composite image may contain amoving object, such as a car or a person) that the user would like toremove from the scene. Additionally, this technique may allow a user toadd objects to an image. In such a scenario, the portions replaced mayrepresent the portion of a scene where the object is to be added. Theimage frame or frames used for the replacement may depict the sceneincluding the object. For example, a user may add and/or remove peopleor other objects while capturing the composite image. In someembodiments, user input may specify areas of the already scanned imageto change. The portable electronic device may capture more image framesof the marked area and update the marked area with new image frames.

Accordingly, it should be appreciated that, while processing a stream ofimage frames representing a scan of an object to be imaged, multipletypes of feedback may be generated and applied to improve quality of theoverall composite image formed. The feedback may be applied to thecomposite image itself, may be supplied to the user or may be applied todevice hardware controlling the imaging conditions. Each of these typesof feedback may be used alone or may be used with other types offeedback in any suitable combination.

Turning to FIG. 1, an example of a system 100 to form a composite imageis illustrated in which some or all of these techniques may be applied.In this example, image frames are captured using a smartphone 102. Itshould be appreciated that techniques described herein may be used withimage frames captured with any suitable portable electronic devicemovable in three dimensions, and a smartphone is used only as an exampleof an image capture device.

As shown schematically in FIG. 1, smartphone 102 can be moved by a user104 in three dimensions to acquire multiple image frames of an object.The object may be a single item, such as a building, or may be apanoramic scene containing multiple items. Accordingly, the term“object” does not imply a limit on the nature of the content of animage.

In this example, the object is a document 106 and the image frames areassembled into a composite image representing a scan of document 106.Document 106 may be any suitable document that user 104 desires to imageusing smartphone 102, such as a page from a book or a magazine, abusiness card, a check for deposit in a bank, a purchase receipt, or anyother type of document. Document 106 may also be held by user 104 orlocated at a distance from user 104, and it is not a requirement thatdocument 106 be placed on surface 108. In this example, the object beingimaged is larger than can be represented in a single image frame whenthe camera of smartphone 102 is zoomed in to acquire image frames withhigh quality. Accordingly, in this example, smartphone 102 is being usedin a mode in which it acquires multiple images of a large object, suchthat these images may be assembled into a composite image. However, itshould be appreciated that some or all of the techniques describedherein may be applied to a single image frame capturing an entire objectwithout the need to form a composite image.

FIG. 2 illustrates components of a smartphone 200 (e.g., smartphone 102in FIG. 1) which is an example of a portable electronic device that maybe used to implement the described techniques. Smartphone 200 mayinclude a camera 202, a display 204, one or more inertial sensors 206and a light source 208. These and other hardware components ofsmartphone 200 may be implemented using techniques as are known in theart. Likewise, software controlling the hardware components may beimplemented using techniques known in the art. Applications 222,however, may include computer-executable instructions that implementimage acquisition and processing techniques as described herein.

Camera 202 may include an imaging sensor which may be any suitable typeof sensor. Camera 202 may include a front-facing and/or a rear-facingcamera, for example.

Light source 208 may be any suitable source of light, such as, forexample, one or more light-emitting diodes (LED). Though, any othertypes of light source may be utilized. Light source 208 may becontrolled to be selectively switched on or off to control motion blurand other parameters.

The inertial sensors 206 may include an accelerometer that tracksrelative motion of the smartphone from one image frame to another, agyroscope that tracks relative motion of the smartphone during a periodof time, a compass, an orientation sensor, and any other types ofsensors that provide an output indicating of a position, orientation ormotion of smartphone 200. Smartphone 200 may also include proximitysensors and other types of sensors.

Smartphone 200 may be moved in three dimensions in any suitable manner,and motion of the device can be detected using inertial sensors 206. Insome embodiments, outputs of the sensors may be captured at times thatare synchronized with capture of image frames. The outputs of sensors206, thus, can be related to what the camera 202 was pointing at when animage frame was acquired. This information provided by the inertialsensors 206 may be used to determine the relative positions of what isdepicted within image frames such that this information may be used todetermine relative positions of image frames within a composite image.

Display, or screen, 204 may be any suitable type of display adapted todisplay image frames as they are being captured by smartphone 200,information comprising feedback to the user and any other information.In some embodiments, display 204 may be an LED-backlit type ofdisplay—e.g., LED-backlit liquid crystal display (LCD) or any other typeof display.

Display 204 may be a touch screen displaying various icons and othercontrols that a user can touch or manipulate in any other manner (e.g.,using gestures). Display 204 may display, in a manner that is perceivedto a user as a continuous live view, image frames of the object beingimaged by camera 202, provide user feedback with respect to controllingimaging conditions and receive user input for controlling operation ofsmartphone 102 while capturing images of the object. In addition,display 204 may include buttons and other components that are adapted toreceive user input.

Operation of each of camera 202, display 204, inertial sensors 206 andlight source 208 may be controlled via one or more controllers. In theexample illustrated in FIG. 2, smartphone 200 includes a cameracontroller 210, a display controller 212, a motion controller 214, and alight source controller 216. These controllers may be implemented usingcircuitry or other suitable components as are known in the art. Though,it should be appreciated that these controllers are shown by way ofexample only, as any type and number of controllers may be included insmartphone 200 and the described techniques are not limited to aparticular implementation of the smartphone. Moreover, the smartphonemay comprise any other controllers, such as a video, audio controller(e.g., multimedia audio controller), and other types of controllers,which may be separate controllers or part of any of the controllersdescribed herein.

Operating parameters of camera 202, display 204, inertial sensors 206and light source 208 may be controlled via respective controllersadapted to transmit control signals to the devices. For example,operating parameters of camera 202, such as the focal length,auto-focus, exposure time, and others, may be controlled via cameracontroller 210. Such a camera controller may be implemented usingcircuitry as known in the art or in any other suitable way. Thesecontrollers may receive commands from processor 218 and provide controlsignals, which implement the command, to associated components.Alternatively or additionally, the controllers may provide informationindicating the state of their associated components.

Light source 208 may be controlled, via controller 216 or othercontroller (e.g., a controller that controls operation of both camera202 and light source 208), to operate in synchronization with camera202. Light source 208 may be, for example, LED-based light source (e.g.,LED “flash”) or other type of light source. The operating parameters ofcamera 202 and light source 208 may be controlled so that smartphone 200may be used to capture images in various environments with differentlighting conditions, including indoors, outdoors at different times ofthe days, such as at dusk or dawn, and at direct daylight. In someembodiments, light source 208 may be controlled to operate in a “torchmode,” which is an operating mode that allows keeping the light on whilecapturing images. In this way, light source 208 may allow takingpictures at night. In some scenarios, operating parameters of lightsource 208 may be controlled by the user. However, in some embodiments,an application executing on processor 218 may determine and/or sendcommands to control operating parameters of any one or more components.

Controller 214 may be used to control operation of inertial sensors 206,including acquiring values from these values. Though a single controlleris shown, it should be appreciated that different inertial sensors(e.g., an accelerometer, a gyroscope, etc.) may have separatecontrollers.

Operating parameters of display 204 may be controlled via displaycontroller 212 to display image frames captured by smartphone 200 andany other information. In some embodiments, display 204 may becontrolled to provide real-time feedback and user guidance. For example,display 204 may be controlled to provide visual guidance to the userwith respect to a manner of obtaining the next image frame in the streamof image frames being captured. When the smartphone is operated to imagea target, display 204 may provide a live camera view showing a live feedfrom camera 202. Controller 212 may also acquire user input, such asinput that may be entered through a touch-sensitive display.

Smartphone 200 also comprises circuitry for performing processing. Inthis example, that circuitry includes a processor 218 and a memory 220coupled to processor 220. Memory 220 may be encoded withcomputer-executable instructions. Memory 220 may be implemented as atleast one computer-readable storage medium that may retain informationfor a sufficient time to provide the computer-executable instructions ina non-transitory form. As used herein, the term “computer-readablestorage medium” encompasses a computer-readable medium that can beconsidered to be a manufacture (i.e., article of manufacture) or amachine.

The computer-executable instructions may be in many forms, such asapplications, or program modules, executed by one or more processors,such as processor 218. Processor 218 may comprise circuitry forexecuting computer-executable instructions.

The computer-executable instructions stored in memory 220, when executedby processor 218, may implement the described image processingtechniques. As shown in FIG. 2, memory 220 may store one or moreapplications 222 for controlling smartphone 200 to implement thedescribed image processing techniques. Applications 222 may comprise oneor more modules for image processing and analysis and forming acomposite image by combining multiple image frames. Applications 222 mayinclude optical character recognition modules, motion estimationmodules, various modules for image pre-processing, reflection and shadowdetection, etc. Some or all of these modules may be executed locally onthe smartphone, independently from any Internet connection. Though, someof the modules may interact with servers or other remote computingdevices such that some or all of the processing described herein may beperformed on those remote computing devices.

In the illustrated example, memory 220 may represent one or more typesof memory, which may be implemented using multiple types of memorycomponents. Applications 222, for example, may be stored in anon-volatile portion of memory 220. A volatile portion of memory 220 maystore other types of data. For example, memory 220 may also store acomposite image 224 formed in accordance with the described techniques,and any other related information, such as information on motion of thesmartphone collected from inertial sensors 206, information obtained asa result of image processing—e.g., results of optical recognitionprocessing, and any other information. Moreover, a composite image onceformed may be moved from volatile to non-volatile memory.

Further, it should be appreciated that memory 220 may store any otherapplications that can be executed on the smartphone. The applicationsmay be downloaded and stored in memory 220, accesses over a network, andreceived in any other manner. One or more of applications 222 may bethird-party applications supported via one or more applicationprogramming interfaces. Memory 220 may also store an operating systemexecuted by processor 218.

FIG. 2 further shows that smartphone 200 comprises battery 226. Itshould be appreciated that smartphone 200 may comprise any othercomponents not shown herein for the sake of brevity, such as wirelesscommunication circuits, input/output components, and any other type ofcomponents. Further, the specific components illustrated are exemplaryof the types of components that may be included in a portable electronicdevice to achieve one or more functions. For example, though battery 226is illustrated, any suitable power source may be present.

FIG. 3 illustrates steps of a real-time processing 300 of image framesto form a composite image using one or more techniques to improve imagequality in accordance with some embodiments. In this example, multipletypes of feedback may be used. Feedback may be generated to guide a userin positioning the smartphone in a way that improves image quality.Alternatively or additionally, feedback may be provided to controllersin the smartphone 200 to impact the conditions under which subsequentimages are captured. Alternatively or additionally, feedback may beprovided to a component that assembles image frames into a compositeimage, to influence the construction of the composite image.

The processing may be implemented on a portable electronic device, suchas smartphone 200 programmed in accordance with techniques as describedherein. Smartphone 102 (FIG. 1) may have multiple operating modes.Different applications, different modules or different portions of anapplication or module may execute to implement each mode. The selectionof a mode may be made based on user input or other conditions that canbe detected by processing on smartphone 200.

In the operating mode illustrated in FIG. 3, a new image frame 302 maybe captured as part of process 300 using a camera, such as camera 202(FIG. 2). Image frame 302 may be acquired as part of acquiring a streamof images that are captured as the camera is being pointed towards anobject. The captured image frames may be used to render a display in anysuitable way. For example, smartphone 200 may operate in a video modeduring which image frames are continuously captured and a live viewcomprising the image frames is displayed to the user.

These captured image frames may be stored in memory for processingand/or later display. The number of image frames stored, and whichspecific image frames are stored, may also depend on user input. Inresponse to one type of user input, for example, a single image framemay be recorded as a still image. Alternatively, multiple image framesin the sequence may be recorded for combining into a composite image ofan object.

To capture image frame 302, a user may point smartphone 102 at an objectdesired to be scanned. Smartphone 102 may then initiate a process ofstoring in memory image frames acquired from the camera upon a userinstruction or automatically. For example, a button may be pressed or avisual feature (e.g., an icon) may be manipulated to instruct smartphone102 to obtain image frames representing a scan of a document or otherobject. Accordingly, though FIG. 3 shows capture of a single image frame302, the depicted processing may be used to capture a sequence of imageframes. One or more aspects of the image capture process may be adjustedover time as successive image frames in the sequence are captured.

Smartphone 102 may be positioned in any suitable orientation withrespect to the object and may be held at any suitable distance from theobject, as embodiments are not limited to any specific way a userpositions and moves the smartphone to scan an object. The object may beof any suitable size, as the described techniques allow obtaining imagesof objects of different sizes, including large objects, by scanningmultiple portions of such objects to capture respective multiple imageframes and combining the image frames into a composite imagerepresenting an image of multiple portion of the object or the entireobject.

Along with acquiring image frame 302, position information for thesmartphone at a time when image frame was taken may be determined basedon outputs of the inertial sensors of the smartphone (e.g., inertialsensors 206 in FIG. 2). As the smartphone is moved to capture images,the inertial sensors may measure position, orientation, and velocity(i.e., direction and speed of movement) of the smartphone. Thisinformation may be used to position image frame 302 within the compositeimage.

As shown at block 304, acquired image frame 302 may be pre-processed toprepare image frame 302 for further analysis. This may compriseimproving quality of image frame 302. The pre-processing 304 may alsoinclude analyzing content of image frame 302 to extract features andobtain one or more parameters. Non-limiting examples of the features maycomprise lines, edges, corners, colors, junctions and other features.Parameters may comprise sharpness, brightness, contrast, saturation,exposure parameters (e.g., exposure time, aperture, white balance, etc.)and any other parameters.

In some embodiments, the pre-processing 304 may involve analyzing theimage frame to determine whether the image frame is suitable for furtherprocessing. This determination may be done as a preliminary analysis,before a quality of the image frame is improved to prepare it for beinginserted into the composite image. If one or more of the parametersobtained by processing image frame 302 indicate that the quality of theimage frame is below a quality required for further processing, imageframe 302 may be excluded from further analysis.

In some embodiments, features extracted from image frame 302 may be usedto determine a sharpness of the image represented in the image framewhich describes the clarity of detail on the image (e.g., a contrastalong edges in the image). It may be determined whether the sharpness ofthe image is below a certain threshold value that may be selected in anymanner. If the sharpness of the image is below the threshold, the imageframe may be discarded.

Furthermore, if a shutter speed of the smartphone camera is slow and theexposure is therefore excessive, the image in image frame 302 may have apoor quality—e.g., may be blurred. Image frame 302 may be excluded fromfurther analysis if it is of an unacceptable quality.

The pre-processing 304 may comprise determining whether to use theacquired image frame in constructing a composite image. Thisdetermination may be made based on, for example, an amount of movementof image frame 302 relative to a preceding image frame. This may bedetermined based on matching the succeeding image frame 302 and thepreceding image frame using respective features of the image frames andmotion information associated with each of the image frames, todetermine an amount of overlap between the image frames.

The motion information may be obtained using measurements collected bythe inertial sensors (e.g., an accelerometer, a gyroscope, etc.) of thesmartphone. The motion of the succeeding image frame may be determinedas a relative motion with respect to a preceding image frame or as anabsolute motion with respect to a reference image frame (e.g., a firstimage frame in the stream of image frames).

If the amount of movement is within a certain range (e.g., in someembodiments, less than 50%), image frame 302 may be used in building thecomposite image. However, the amount of movement that is above a certainthreshold value (e.g., in some embodiments, greater than 50% relative toa prior image frame) may be taken as an indication that the smartphoneis moved out of a certain range within a distance from a position atwhich a preceding image frame was captured and a position at which thesucceeding image frame was captured. In this case, the image frame maybe discarded.

Furthermore, if the amount of movement of the image frame is below athreshold value (e.g., in some embodiments, less than 2%), it may betaken as an indication that the smartphone was not moved from a timewhen the preceding image frame was captured and a time when thesucceeding image frame was captured. If it is determined that thesucceeding image frame was not displaced relative to the preceding imageframe and is therefore a redundant image frame, the succeeding imageframe may be discarded. It should be appreciated that acceptablethreshold amounts used to determine an absence of movement or anexcessive amount of movement may be selected in any suitable manner andmay vary in different embodiments.

Regardless of the way in which it is determined whether image frame 302is to be discarded or whether it can be used further, image frame 302may be discarded if it is determined to be not acceptable for furtherprocessing (not shown in FIG. 3).

If it is determined that image frame 302 is of an acceptable quality forbeing included in a composite image, a quality of image frame 302 may beimproved before inserting the image frame into the composite image.

Because the smartphone may acquire image frames representing the objectat different orientations as the user moves the device in threedimensions, rotation of image frame 302 relative to a prior image framemay be detected. The pre-processing 304 may involve unrotating featuresin image frame 302 or otherwise translate the image frame into anotherframe of reference to align image frame 302 with the prior image frame.

In some embodiments, the pre-processing 304 may also comprise improvingquality of image frame 302 by performing undistortion of an imagerepresented in image frame 302 to correct for lens distortion,correcting for warping of the image, smoothing the image, correcting forwhite balance and performing any other suitable processing of imageframe 302.

Next, pre-processed image frame 302 may be inserted (306 in FIG. 3) intothe composite image, interchangeably referred to herein as a graph map.In the embodiment illustrated, a graph map may be a data structurestored in computer memory representing relative positions of imageframes within a composite image. A representation of the composite imagemay be maintained and updated in the memory of the smartphone (e.g.,memory 220 in FIG. 2) as multiple image frames are combined in thecomposite image. In some embodiments, the graph map may be arepresentation of the composite image such that, when an image isdisplayed, it is rendered from the graph map in conjunction with otherinformation indicating which portions of the graph map are to bedisplayed. In other embodiments, the composite image may be stored asvalues for pixels in an image, which may be directly rendered on adisplay, or in any other suitable format. Alternatively, as each newimage frame is integrated into a composite image, it may change thevalues associated with the pixels. Accordingly, insertion of an imageframe into the composite image may be performed in any suitable way,including by integrating visual information acquired from the imageframe into a representation of the composite image or a data structurefrom which the composite image is rendered.

In some embodiments, preprocessing may determine whether to insert animage frame into the composite image. For example, image frame 302 maybe inserted into the composite image when image frame 302 overlaps to anacceptable degree with a prior image frame. The prior image frame may bean image frame immediately preceding the succeeding image frame 302 orother prior image frame.

Image frame 302 may be combined with other image frames in the compositeimage based on the features identified for the image frame which may beextracted during the pre-processing 304. The features, combined withpositional data determined for image frame 302, may be represented aspoints in a three dimensional point cloud. Processing circuitry of thesmartphone may maintain an association between the points in the cloudand image frame 302 from which they were extracted.

In some embodiments, described in more detail below, image frame 302 maybe represented as a set of points in the point cloud and may beinitially positioned within the point cloud based on the positioninformation of the smartphone at the time image frame 302 was captured.Image frame 302 may be positioned within the point cloud based on aposition of a prior image frame within the point cloud.

Once image frame 302 is inserted into the composite image, the compositeimage including the image frame may be adjusted, as shown at block 308in FIG. 3. The adjustment may comprise processing the composite image toimprove its quality. Any one or more techniques may be applied at block308 to adjust the graph map storing the data representing the compositeimage.

The adjustment at block 308 may be based on projecting points associatedwith multiple image frames in the point cloud to a common referenceplane representing the composite image. When the portions of the objectbeing imaged represented in different image frames overlap, adjacentsets of points may include points corresponding to the same imagefeatures. The three dimensional positions of sets of points may beadjusted so that the points representing the same features overlap inthe reference plane. In this way, a coarse alignment of image frames,associated with the sets of points, may be achieved.

Accordingly, image frame 302 may be coarsely positioned by matching theset of points representing the image frame with respect to one or moresets of points representing previous overlapping image frames (e.g.,image frames captured prior to the current image frame).

The quality of the composite image may be further improved by a fineralignment of the sets of points each representing an image frame in thepoint cloud. Such finer adjustment may be performed to reduceinconsistencies based on “global” positioning of image frames. Globalpositioning may involve positioning an image frame within the compositeimage based on positioning of image frames beyond the immediatelypreceding image frame. The finer alignment may involve adjustingrelative position and orientation of the sets of points to reduceinconsistencies resulting, for example, from errors in inertial sensoroutputs that accumulate as the smartphone is moved back and forth,nearer and further from the object being imaged.

Inconsistencies may also result from an accumulation of small errors inalignment of one set of image points to the next as part of the coarsealignment. As a set of points extracted from each incoming image frameare added to the point cloud by coarsely positioning the set relative toprior image frames, an image frame may become aligned to more than oneprior image frame. The stream of image frame may be thus taken as closedin a “loop.” When the “loop closing” is detected, an inconsistencybetween the position of the image frame in different alignments mayexist. The fine alignment may reduce this inconsistency to achieve amore accurate mapping between the image frames.

Further improvement of the quality of the composite image may beachieved by using image fill techniques that allow avoiding distractingfeatures in the composite image. For example, a user finger, which mayappear on an image of an object being held by a user, may be removedfrom the image and a corresponding area may be filled with contentsimilar to that in other areas of the image.

It should be appreciated that the quality of the composite image may beimproved in various other ways, including by selecting which imageframes or portions of image frames to use in rendering a compositeimage. In some embodiments, processing at block 308 may entailidentifying portions of the data captured from a sequence of imageframes to omit from the composite image or to replace with other data.As an example, processing at block 308 may identify that the objectbeing imaged includes undesirable items. Portions of image framesdepicting those undesirable items may be removed or replaced in thecomposite image. As a specific example, in the scenario illustrated inFIG. 1, user 104 may be holding document 106 with a finger. That fingermay appear in the composite image, but processing at block 308 mayremove it from the composite image. A technique for processing toidentify a finger and remove it from an image is described below.

After the composite image is adjusted at block 308, process 300 mayfollow to block 310 where the quality of the composite image may bechecked and improved. This may be performed in real-time, as imageframes of the object being scanned are being captured. The process ofquality checking and improving may comprise identifying areas ofdifferent quality in the composite image. This may include selectingfrom among multiple image frames to provide details of one or moresegments of the composite image. In some embodiments, techniques may beemployed to identify relative quality levels of image frames from whichinformation about the same segment may be obtained. Using relativequality information, information from one or more of multiple imageframes representing the same segment may be identified and used inrendering the composite image.

Image quality as it relates to an overall image or one or more imageframes combined into a composite image may be determined in any one ormore suitable ways. In some embodiments, image frames used in renderinga composite image are stored in memory such that each can be separatelyprocessed or adjusted before being used to render the composite image.However, there is no requirement that the processing at block 310 beperformed on entire image frames or single image frames. Any suitableportion of the image data acquired may be processed to determine imagequality and adjusted to improve image quality.

As a specific example, processing at block 310 may involve determiningthe relative image quality based on quality of optical characterrecognition (e.g., when the object being imaged is a document withcharacters, the likelihoods associated with identifying characters mayindicate image quality), presence of reflections or shadows, and otherartifacts. In this way, segments of low quality may be identified andreplaced by other image segments, depicting the same portions of theobject being imaged with a higher quality. The quality of the compositeimage may be improved in any other manner, as the described techniquesare not limited in this respect.

Next, process 300 may provide the composite image (312) as part of anoutput. The output, for example, may be directed to a display of theportable electronic device such that, as the composite image isconstructed and adjusted, the evolving image will be displayed inreal-time to a user. Though, other uses may be made of the output. Forexample, the composite image may be stored in memory of the smartphone(e.g., memory 220 in FIG. 2).

The composite image may be rendered on the display of the smartphone inany suitable manner and may be updated and adjusted as scanning of theobject progresses. Regardless of the number and nature of alignmentprocesses, processing circuitry may maintain an association between thepoints in the point cloud and the image frames from which they wereextracted.

Once an image of the object is completed, the image may be used in anysuitable way. For example, it can be displayed, stored in the memory ofthe smartphone, printed out, transmitted via a network (e.g., in theemail message), provided to an application, shared with othersmartphones (e.g., via wireless image sharing), and used in any othermanner.

Forming the composite image in accordance with some embodiments mayinclude analyzing portions of the composite image as it is being formedand using results of the analysis to guide acquisition of further imageframes to complete the composite image. Accordingly, as shown in FIG. 3,process 300 may include providing a real-time (“live”) feedback (314) tothe user of the smartphone. Techniques as described herein to identifysegments of the composite image of low quality may be executed inreal-time—while the user is moving the smartphone to acquire an image ofan object. This real-time identification of low-quality segments may beused to render a display indicating to the user areas of the object thatshould be imaged again to improve image quality. Such areas may beindicated to the user in any suitable manner. For example, a coloredframe may be displayed emphasizing the area that needs to be reimaged.

When the object being imaged is a sheet of paper, the size of the papermay be determined automatically such that the extent of the sheet ofpaper that has been imaged may be compared to the detected page size,allowing portions of the page that have not been imaged to beidentified. Feedback to the user may direct the user to image again theportions of the page that have not been imaged.

Additionally or alternatively, it may be indicated to the user in asuitable manner in which way to position the smartphone to captureadditional images of an object being imaged. For example, positioninformation may be output to the user to alter the orientation of thesmartphone to avoid reflections or other image artifacts. Such positionmay indicate a direction or orientation of the smartphone to avoidcreating image artifacts.

As another way of providing feedback, process 300 may comprisecontrolling settings of hardware that capture subsequent image frames.FIG. 3 shows that process 300 includes setting capture strategy for anext image frame in the stream of image frame, shown by block 316 inFIG. 3. Setting the capture strategy may include adjusting one or moreoperating parameters of one or more hardware components of thesmartphone, such as camera 202, display 204, inertial sensors 206, lightsource 208 and any other component which can be controlled to operate tocapture images of the imaged target. As a specific example, commands maybe sent to camera controller 210 to adjust the zoom or focus of thecamera. Each of these hardware components may be controlled via acorresponding controller—e.g., controllers 210, 212, 214 and 216 in FIG.2—or any other type of controller.

Alternatively or additionally, process 300 may entail adjustingprocessing of one image frame based on a prior image frame. In theexample of FIG. 3, feedback is shown provided to pre-processing block304. This feedback, for example, may be applied to select parameters toadjust during pre-processing or the amount of adjustment of one or moreparameters.

In some embodiments, feedback on the image processing may be provided invarious ways based on processing of an image frame and analysis of thecomposite image. FIG. 4 illustrates an overview of an exemplary process400 of providing feedback in accordance with some embodiments. Process400 may be implemented by a processor within a smartphone or may beperformed under the control of any suitable processor.

Process 400 may start, for example, when a smartphone operated to scan ascene captures an image frame comprising an image of the scene. Thescene may comprise any types of object, made up of any number and typeof items, and may be imaged as the smartphone is moved in differentorientations in three dimensions. An image of the object being imagedmay be displayed as a real-time live view. Motion sensors of thesmartphone may provide output indicating a position of the smartphone ata time when the image frame was captured.

At block 402, the captured image frame may be processed, which mayinclude extracting various features from the image frame, unrotatingsome or all of the features as needed, correcting for lens distortion,smoothing, white balancing, and performing other types of pre-processingto correct one or more characteristics of the image frame. In someembodiments, it may be determined whether to discard the image framebecause its quality is below a minimum acceptable requirement, orwhether to proceed with using the image frame as part of building thecomposite image. In the example illustrated, the image frame isdetermined to be acceptable for further processing.

Next, at block 404, the processed image frame may be incorporated into arepresentation of the composite image. As discussed above, therepresentation may comprise a three dimensional point cloud comprisingsets of points representing features of respective image frames. Theimage frame may be coarsely positioned within the point cloud based on aposition of a preceding image frame positioned within the point cloud.The content of the representation of the composite image may then beanalyzed, at block 406. The analysis may include determining quality ofthe composite image.

Based on the quality analysis, one or more corrective actions may bedetermined to improve the quality of the composite image, at block 408.The corrective actions may comprise reducing inconsistencies in thecomposite image to finely align image frames within the composite image.Multiple others corrective actions may be performed.

Feedback based on the corrective action may then be generated, at block410. The feedback may comprise indicating to the user that a portion ofthe composite image needs to be imaged again, that a portion of theobject which was imaged has a poor quality, etc. The feedback may begenerated in real-time, as the smartphone is being used to image thescene.

In some embodiments, the feedback may be provided so that the user mayadjust positioning of the smartphone while imaging the scene. Feedbackto the user may be provided in any suitable way. In some embodiments,that feedback may be provided through a user interface of the smartphone200. A suitable user interface may include, for example, a display,through which the feedback may be displayed graphically or as text, oran audio output, through which the feedback may be presented in anaudible fashion.

In some embodiments, an instruction to adjust one or more operatingparameters of a camera of the smartphone, including its position,orientation or rotation, may be generated. The instruction may beprovided as a visual indication to the user of the smartphone or may begenerated automatically as a control signal to one or more hardwarecomponents of the smartphone.

Process 400 may then follow to block 412 where it may be determinedwhether there are more image frames to be captured. For example, as thesmartphone is used to scan a scene, multiple images of differentportions of the scene may be captured until a user provides inputrepresenting a stop command. Accordingly, process 400 may executecontinuously as new image frames are added to the composite image andthe composite image is adjusted based on the new frames. While the scancontinues, a next image frame may be acquired and process 400 may loopback to block 402. Each new frame may be used to expand the extent ofthe object represented in the composite image. Alternatively oradditionally, as image frames are captured that depict portions of anobject already depicted in captured image frames, the new image framesmay be combined in any suitable way with the previously captured imageframes in the overall composite image. If the image acquisition iscompleted, process 400 may end.

FIG. 5 illustrates another process 500 of providing feedback to the userof the smartphone based on real-time processing of image frames capturedas part of a scan of an image. Process 500 includes pre-processing of acaptured image frame, at block 502, and incorporating the processedimage frame into the representation of the composite image, at block504, which may be performed similar to processing at blocks 402 and 404in FIG. 4. As with process 400, process 500 may be performed undercontrol of a processor of a smartphone executing storedcomputer-executable instructions or using other suitable circuitrywithin a smartphone, or in any other suitable way.

As shown in FIG. 5, at block 506, quality of a depiction of a scene inthe representation of the composite image may be determined. Thisanalysis may involve analyzing the content in one or more portions ofthe composite image for characteristics representative of a reflection,a shadow, or other artifacts. The quality analysis may also compriseanalyzing a result of applying an optical character recognition to oneor more portions of the composite image.

The determination of the quality of the depiction of the scene in therepresentation of the composite image may include analyzing imagefeatures in an image frame and image features in one or more prior imageframes representing overlapping portions of the scene. The imagefeatures may comprise a specular highlight and other features. If one ormore image features compromising quality of the image frame are detectedin the image frame, one or more characteristics of the image frame maybe corrected before incorporating the image frame into the compositeimage.

At block 508, a position parameter of the smartphone may be computed.Because the smartphone may be moved in an unrestricted manner in threedimensions and can be used to capture images of a scene at differentdistances and different angles relative to a plane of the scene, thecomputed position parameter may comprise at least one position parameterthat does not define a location within a plane parallel to a plane ofthe scene, at least one position parameter that defines a spacingbetween the scene and the smartphone, at least one position parameterthat defines an angle of the smartphone with respect to a normal to thescene, and/or other type of parameter, such as a parameter indicatingwhere to position the smartphone to acquire subsequent image frames.

Next, feedback to adjust positioning of the smartphone based on thecomputed position parameter may be generated, at block 510. The feedbackmay include guidance to the user with respect to further operation ofthe smartphone to capture images of the scene. For example, when aportion of the scene has not been imaged yet or has been imaged to yieldlow quality images, an indication to position the smartphone to rescanthat portion of the scene may be provided as a feedback. Any other formsof the feedback may be provided additionally or alternatively. Thefeedback may be provided in real-time, while the image frames of thescene are acquired.

Process 500 may then follow to block 512 where it may be determinedwhether there are more image frames to be captured, which may be thecase when the scan of the scene is not yet completed and the usercontinues to operate the smartphone to capture images. While the scancontinues, a next image frame may be acquired and process 500 may loopback to block 502. If the image acquisition is completed (e.g., if userinput was detected instructing the smartphone to stop the scan), process500 may end.

In some embodiments, operation of a camera of a smartphone (e.g., camera202 of smartphone 200 in FIG. 2) may be controlled based on processingof images captured by the camera, and analyzing and improving a qualityof the images, as a composite image is being built. The camera may becontrolled, for example, via a controller such as controller 210 (FIG.2).

FIG. 6 illustrates generally a process 600 of forming a composite imagefrom multiple image frames representing images of a scene acquired usinga smartphone. As with processes 400 and 500, process 600 may beimplemented under the control of a processor on the smartphone or inother suitable processing circuitry. Process 600 may comprise sequentialprocessing of image frames as they are captured and added to thecomposite image. It should be appreciated that, though shown as separateprocesses, the processes 400, 500 and/or 600 may be performedconcurrently as a sequence of image frames is being captured. Theseprocesses may be performed on separate processing cores or may beperformed on a single processing core using a time multiplexingarrangement.

As illustrated in FIG. 6, process 600 may include pre-processing of thecaptured image frame, at block 602, and incorporating the pre-processedimage frame into the representation of the composite image, at block604, which may be performed similar to processing at blocks 402 and 404in FIG. 4. In process 600, sequentially processing image framescomprises, for image frames captured after controlling the camera tooperate with the determined operating parameter, prior to incorporatingan image frame in the representation of the composite image frame,adjusting the image frame based on the determined operating parameter ofthe camera.

Next, at block 606, a quality of depiction of the scene in a portion ofthe representation of the composite image may be determined. Thedetermined quality of depiction of the scene may be expressed as a valueof a metric.

As shown in FIG. 6, at block 608, an operating parameter of a camera ofthe smartphone may be selectively determined based on the determinedquality of the depiction of the scene. In some embodiments, determiningthe operating parameter may comprise activating a flash on the camerawhen the value of the metric is below a threshold. Additionally oralternatively, selectively determining the operating parameter maycomprise adjusting the amount of data captured in the image frame whenthe value of the metric is below a threshold.

The camera of the smartphone may then be controlled to operate with thedetermined operating parameter, at block 610.

Process 600 may then follow to block 612 where it may be determinedwhether there are more image frames to be captured, which may be thecase when the scan of the scene is not yet completed and the usercontinues to operate the smartphone to capture images. While the scancontinues, a next image frame may be acquired and process 600 may returnto block 602. If the image acquisition is completed (e.g., if user inputwas detected instructing the smartphone to stop the scan), process 600may end.

When a smartphone or any other mobile device is used to capture andprocess multiple image frames in accordance with some embodiments, thedevice may be moved freely back and forth, and closer and nearer andfurther from a scene being imaged. As a result, the image frames may beacquired in different planes and oriented differently with respect tothe each other and a plane of the scene. In this way, the scene orportions of the scene may be captured in image frames positioned inplanes that are not parallel to each other or parallel to the plane ofthe scene. More generally, the frame of reference of each image framemay be different.

Accordingly, to account for the three dimensional space in which imageframes are acquired, image frames may be processed to map the imageframes into a common frame of reference. In this common frame ofreference, the relative positions of the image frames may be determinedor adjusted. The positions within the common frame of reference maydefine the position of the image frames within the composite image. Inthe embodiment illustrated in FIG. 3, the positions within the commonframe of reference may initially be used to insert an image frame into agraph map, such as at block 306.

In some embodiments, mapping of an image frame may be performed based onfeatures in the image frame. Features, such as corners or bright points,may be identified using known image processing techniques. The positionsof these features within the common frame of reference may initially beestimated based on sensor outputs that provide an indication of motionand/or orientation of the smartphone.

Various factors may be considered in mapping features from an imageframe to the common frame of reference. An orientation of thesmartphone, distance from the object being imaged, zoom of the camerawithin the smartphone and any other sensor output providing positionalinformation. This positional information may be used to compute alocation of portions of an object being imaged at the time the imageframe was acquired. This information may be used to translate featuresof an image frame to the common frame of reference.

In some embodiments, the features within the image may be depicted aspoints, such that the features of the image frames may collectively beregarded as defining a three dimensional point cloud. Such a pointcloud, representing multiple image frames and their relationships, isshown in FIG. 7. The point cloud may be represented, such as by datacharacterizing it that is stored in a computer memory, to maintain anassociation between image frames and sets of points representingfeatures extracted from the image frames. Moreover, the point cloud maybe maintained in a way that allows the relative position and orientationof the points associated with an image frame to be adjusted.

As discussed above, when an image frame is captured, processing of theimage frame includes extracting features. The features may also beprocessed to improve subsequent feature matching. Each image frame maybe represented as a set of points representing features extracted fromthat image frame. FIG. 7 illustrates schematically exemplary imageframes 702, 704 and 706 each representing a portion of a scene andacquired as the scene is being scanned by a smartphone. In this example,image frames 704 and 706 represent different portions of the sameobject, a tree.

Once features from each of the image frames 702, 704 and 706 areextracted, the image frames may be associated with sets of pointsrepresenting the features in a three dimensional point cloud space 709.The point cloud space may be represented in any suitable way, but, insome embodiments, is represented by data stored in computer memoryidentifying the points and their positions. As shown in FIG. 7, imageframe 702 may be represented as a set of points 708, image frame 704 maybe represented as a set of points 710, and image frame 706 may berepresented as a set of points 712. Image frame 702 includes, forexample, points 714 and 716 representing corresponding image features ofthe image (a car) represented in image frame 702. Other points in imageframe 702 and points in image frames 704 and 706 are not labeled for thesake of simplicity of the representation.

The sets 708, 710 and 712 may initially be positioned within the pointcloud 709 based on position information of the smartphone at the timethe associated image frame was captured. Though, subsequent processingmay adjust the positioning of the sets within the point cloud space tocreate a composite image that more accurately represents a scene beingimaged. Alternatively or additionally, one or more of the sets may bedeleted or altered, also as a result of processing to provide a desiredcomposite image.

An image matching approach may be used to adjust the relative positionsof the sets of points. In acquiring a sequence of image framesrepresenting a scan of an object, the smartphone may be operated tocapture image frames at a sufficient rate that successive image frameswill at least partially overlap. By identifying overlapping portions ofimage frames that are adjacent in the sequence, the relative position ofthose image frames may be adjusted so that those features align.

Prior to matching the points representing features in adjacent images,the points may be associated with a common frame of reference forcomparison. A common frame of reference, allowing comparison andalignment between successive image frames, may be created by projectingthe points associated with each image frame into a common plane. Pointsin sets of points 708, 710 and 712 may be projected, as shown by dashedlines 718 in FIG. 7, into a common plane of reference 720. Common planeof reference 720 may represent the two-dimensional composite image thatis being built as the scan progresses.

In some embodiments, as each set of points is projected, itsthree-dimensional position may be adjusted to ensure consistency withsets of points containing points representing an overlapping set offeatures. For example, as shown in FIG. 7, set of points 710 and set ofpoints 712 may include an overlapping set of features, such as a featurerepresented by a point 722. This point may correspond to the samefeature in images 704 and 706. Coarse alignment of image frames inaccordance with some embodiments may comprise adjusting a threedimensional position associated with each set of points to achievecoincidence in plane 720 between points representing the same features,which may improve quality of the composite image rendered using thosethree dimensional positions for the image frames to assemble thecomposite image.

As more image frames are gathered and additional sets of points areadded to the point cloud, the relative position and orientation of thesets of points may be adjusted to reduce inconsistencies, to thusachieve global alignment of the image frames. Thus, sets of points inpoint cloud 709 may be repeatedly adjusted and projected to plane 720 toform the composite image that is displayed to the user on a userinterface, such as the user interface on a display of a camera of thesmartphone, or otherwise processed.

FIG. 8 illustrates a process 800 of building a composite image byrepresenting features of image frames in the three dimensional pointcloud, in accordance with some embodiments. Process 800 may beimplemented by an application executing on a processor a smartphone, asdescribed above, or using any other suitable processing circuitry.

Process 800 may start at block 800 when capturing a stream of imageframes of an object by a smartphone (e.g., smartphone 102 in FIG. 1) isinitiated. The capturing may be initiated based on user input or anyother type of trigger. In some embodiments, the smartphone may beinstructed to operate in a capture mode to acquire image frames using acamera (e.g., camera 202 in FIG. 2). The object may be any object whichcan be imaged as the smartphone is moved in the three-dimensional space.

An image frame may be acquired at block 804. Next, the acquired imageframe may be processed by computing that extracts one or more imagefeatures from the image frame. The features may be any suitable types offeatures, such as color, shape, texture features, etc. For example,lines, edges, corners, contours, junctions and any other features may beextracted. A subset of the extracted features may be selected at block808. In some embodiments, this selection may involve optimizing thefeature distribution to achieve approximately the same number offeatures in each image frame from the image frames that form a compositeimage, where that number of features may be independent of texturecharacteristics of the image frames. However, any suitable approach forselecting suitable features for use in matching image frames may beapplied.

As shown in FIG. 8, next, at block 810, process 800 may find featurecorrespondences between pairs of image frames. A succeeding image framemay be compared with one or more previously captured image frames toidentify corresponding features between each pair of frames. Suchcorrespondences may be identified based on the nature of the featurecharacteristics of the image frames surrounding the features or othersuitable image characteristics.

In some embodiments, the processing at block 810 may involve using a setof features computed for a respective image frame to estimate theepipolar geometry of the pair of the image frames. Each set of featuresmay be represented as a set of points in a three-dimensional space.Thus, the image frame acquired at block 804 may comprisethree-dimensional points projected into a two-dimensional image. When atleast one other image frame representing at least a portion of the sameobject, which may be acquired from a different point of view, has beenpreviously captured, the epipolar geometry that describes the relationbetween the two resulting views may be estimated. The epipolar geometrymay be estimated using techniques as are known in the art.

In some embodiments, identification of feature correspondences mayinclude searching, for each point in the image frame, for acorresponding feature in another image frame along a respective epipolarline. The three-dimensional points representing the image frame may bere-projected to establish correspondences to points that may be visiblein the current image frame but not in the immediately preceding imageframe—e.g., when the current image frame overlaps with a prior imageframe other than an immediately preceding image frame in a stream ofimage frames.

At block 812, an initial pose of a camera indicating its position andorientation with respect to an object being scanned at a time when anassociated image frame was acquired may be estimated, at block 812. Theinitial pose may be estimated based on output from inertial sensors(e.g., sensors 206 in FIG. 2) and/or any other types of sensors of thesmartphone. It should be appreciated that the initial camera pose may beestimated simultaneously with processing at one or more of the blocks804-810.

After the image frame is inserted into a composite image, the initialpose of a succeeding image frame may be adjusted based on poses of imageframes that are already present in the composite image and at leastpartially overlap with the succeeding image frame. In some embodiments,the adjustment may be performed for the entire composite image. Though,in some embodiments, poses of a portion of image frames in the compositeimage may be adjusted. In some embodiments, the simultaneous adjustmentof poses of a number of overlapping images may be referred to as bundleadjustment.

In some embodiments, initially, the set of points representing featuresextracted from the image frame, may be positioned within athree-dimensional point cloud based on position information of thesmartphone at the time the associated image frame was captured, such asthe estimated pose of the smartphone. As each set of points is added tothe point cloud, its three-dimensional position may be adjusted toachieve coincidence in the plane between points representing the samefeatures, thereby improving the quality of the composite image. In thisway, a coarse alignment of the image frames may be performed.

The coarse alignment is based on a local comparison of an image frame toone or more easily preceding image frames that were acquired in asequence of frames. As more image frames in a sequence of image framesare processed, additional information becomes available to refine thecoarse estimation of the relative positions. Accordingly, it may next bedetermined, at decision block 814, whether the number of image framesthat have been captured is greater than a threshold n. If it isdetermined that the number of image frames is greater than thethreshold, process 800 may follow to block 816 where the initial pose ofone or more image frames may be adjusted. The adjustment may beperformed by solving an optimization problem, such as, for example, abundle adjustment, or other type of problem. Bundle adjustmentalgorithms can simultaneously solve for locations of all of the camerapositions to yield globally consistent solutions. The bundle adjustmentor other suitable techniques may be used to generate the point cloudcomprising sets of points each representing features extracted from animage frame.

If it is determined that the number of image frames is smaller than thethreshold, process 800 may branch to decision block 818 where it may bedetermined whether one or more features of the succeeding image frameoverlap with at least one prior image frame other than the immediatelypreceding image frame. If this is the case, a “loop closure” may bedetected, at block 820. An example of a loop closure, in atwo-dimensional space, is illustrated in FIG. 15, below. When a loopclosure is detected, the three-dimensional points may be re-projectedinto a current viewpoint so that the three-dimensional positions andcamera poses can be optimized. When no loop closure is detected based onthe features overlap, process 800 may follow to block 822, as shown inFIG. 8.

At decision block 822, it may be determined whether the scanning of theobject is to be stopped. The scanning may be stopped based on userinput, passage of a predetermined time, determination that image datafor the area representing an object is required, or in any other manner.If it is determined that the scanning of the object is completed,process 800 may follow to block 824, where a surface of the scannedobject may be reconstructed using the three-dimensional point cloud. Thesurface reconstruction may include de-warping, refection removal andother adjustments to improve the quality of the composite image. In thisway, a geometry of the scanned object may be determined. The positionsof the acquired image frames relative to that geometry have also beendetermined. An image of the scanned object may then be rendered, atblock 826. Because the geometry of the scanned object has beendetermined, a viewpoint of the output image can be determined.

When it is determined, at decision block 822, that the scanning of theobject is not completed, process 800 may branch back to block 804 wherea next image frame may be acquired by scanning the object.

In the embodiments described above, image frames representing images ofan object being imaged may be captured by a smartphone moved, duringscanning process, into multiple orientations in the three dimensionalspace. However, by mapping features representing the image frames into acommon plane of reference, processing may be performed in a commonreference plane much the same way that processing might be performed onimage frames acquired by a portable device moving with a singleorientation. In the following, processing in a common reference plane isdescribed for ease of illustration. It should be remembered, however,that when adjusting estimates of the relative position between imageframes, the possibility of motion in all dimensions in thethree-dimensional space may be accounted for in the adjustments.

To further illustrate the processes of coarse image frame alignment andsubsequent refinement, FIG. 9 provides an example of coarse positioningof two consecutive image frames in accordance with some embodiments.Coarse positioning of image frames of a scanned object may comprisealigning consecutive image frames based on matching portions of theimage frames showing corresponding portions of the object being scanned.FIG. 9 schematically illustrates such a process of aligning two imageframes based on matching portions of the image frames corresponding torespective portion of the object being scanned. In this example, animage frame 900 represents a preceding image frame and image frame 902represents a succeeding image frame taken as a scanning device movesover the object being scanned. Though, image frame 902 may be alignedwith any one or more image frames that partially overlap with imageframe 902, based on matching content of the image frames within theoverlapping areas.

During the coarse positioning, an initial pose of image frame 902 mayfirst be estimated based on information from one or more inertialsensors (e.g., inertial sensors shown in FIG. 2). The initial poseestimate may be associated with some imprecision expressed as a zone ofuncertainty 903, as shown in FIG. 9. Though not readily illustrated in atwo dimensional drawing, the zone of uncertainty may representuncertainty in both displacement HI and orientation. In a scenario inwhich image frames are captured using a portable electronic device,uncertainty and orientation may reflect the possibility that theportable electronic device has been rotated in the plane parallel to theplane of an object being imaged as well as tilted in any number ofdirections with respect to that plane.

In some scenarios, the zone of uncertainty may be small enough that aninitial pose estimate may provide adequate coarse positioning of imageframe 902. However, in some embodiments, alternatively or additionally,a second coarse positioning technique based on matching content in aportion of image frame 902 with content in a corresponding portion ofimage frame 900 may be used.

The pose of image frame 902 that results in a suitable match of contentin the overlapping areas may be taken as the position of image frame 902relative to image frame 900. The pose that provides a suitable match maybe determined based on aligning features or other image content.Features, such as corners, lines and any other suitable features, may beidentified using known image processing techniques and may be selectedfor the matching in any suitable way.

In some embodiments, the matching process may be simplified based onpositioning information. It may be inferred that the pose of image frame902 that aligns with image frame 900 provides a pose within area ofuncertainty 903. To reduce processing required to achieve alignment andto thus increase the speed of the local positioning of image frames, insome embodiments, the position information obtained from the inertialsensors may be used. If image frame 902 in aligned with image frame 900using feature matching, processing required to find correspondingfeatures can be limited by applying the zone of uncertainty 903. Forexample, image frame 900 includes a feature 910. A corresponding featureshould appear in image frame 902 within a zone of uncertainty 903Aaround a location predicted by applying position information output bythe inertial sensors that indicates motion of the smartphone between thetimes that image frame 900 was acquired and image frame 902 wasacquired. Accordingly, to find a feature in image 902 corresponding tofeature 910, only a limited number of features need to be compared tofeature 910.

It should be recognized that feature matching as shown in FIG. 9 isillustrated based on features already projected into a common plane forboth image frames being compared. The projection of each image frameinto the common plane of reference is based on a mapping derived from anassumed position and orientation of the portable electronic device whenimage frame was captured. The assumed orientation may impact the spacingbetween features and other aspects of the image frames as projected intothe common plane of reference. Inaccuracies in the assumed orientationsmay impact how well features in one image frame align with acorresponding set of features in another image frame when both areprojected into the common reference claim. Accordingly, searching for arelative position and orientation of image frames that aligncorresponding features in the image frames may entail determining theappropriate orientation of the portable electronic device used inprojecting feature sets of the image frames into the common plane ofreference.

If other matching techniques are employed, position information may alsobe used in a similar way. For example, overlapping regions in differentposes of image frame 902 are iteratively compared on a pixel-by-pixelbasis, the position information can be used to identify overlappingportions to be compared and to limit the number of poses to be tried tofind a suitable match.

Regardless of the matching technique employed, any suitable criteria canbe used to determine a suitable match. In some embodiments, a match maybe identified by minimizing a metric. Though, it should be appreciatedthat a suitable match may be determined without finding an absoluteminimum. As one example, a pose of image 902 may be selected by findinga pose that minimizes a metric expressed as the sum of the difference inpositions of all corresponding features. Such a minimum may beidentified using an iterative technique, in which poses are tried.Though, in some embodiments, known linear algebraic techniques may beused to compute the pose yielding the minimum.

In FIG. 9, image frames 900 and 902 contain matching portions comprisingequal image content which is shown by way of example only as a strawman.Once the equal image content in image frames 900 and 902 is identifiedusing any suitable technique, the image frames may be aligned using theequal image content. In FIG. 9, image frame 900 aligned with image frame902 is shown by way of example only as image frame 902A.

In embodiments of the invention, scanning of an object may be performedby moving a smartphone over the object. A stream of image frames maythus be captured which are then stitched together to form a compositeimage representing the object. As a user is moving the portableelectronic device relative to the object and new image frames in thestream are being captured, their respective coarse positions may bedetermined. Each coarsely positioned image frame may be presented on adisplay device in a position proportional to its determined positionwithin the composite image. The coarse positioning can be performed fastenough that image frames may be displayed to the user on the displaydevice with a small delay relative to when the image frames arecaptured. As a result, a composite image representing a progression ofthe scanning process of the object being scanned appears to be paintedon the display device. Furthermore, a fine adjustment may be made to therelative positions of the coarsely positioned image frames.

FIGS. 10A-D illustrate a process of scanning an object by capturing astream of successive image frames of the object, in accordance with someembodiments of the invention. In these examples, the object beingscanned comprises a text document 1000. As the scanning device, whichmay be a smartphone with a camera as described above, moves over theobject, images of the object are captured at intervals, which areillustrated to be periodic in this example, thus resulting in a sequenceof image frames. Each succeeding image frame may be initially positionedbased on a respective preceding image frame to obtain an estimate of aninitial pose of the succeeding image. As described above, positioninformation representing movement and orientation of the scanning deviceobtained from the inertial sensors may be used to simplify theprocessing.

The image frames are shown in FIGS. 10A-D as superimposed over textdocument 1000 to demonstrate exemplary movements of the scanning devicerelative to the text document. It should be appreciated that eachsubsequent image frame may be oriented in any suitable way with respectto a preceding image frame as embodiments of the invention are notlimited to any particular movement of the scanning device over an objectbeing scanned. In the embodiment illustrated, an image frame ispositioned based on comparison to an immediately preceding image frame,which is not a requirement of the invention. A succeeding image may belocally positioned by being aligned with respect to any other precedingframes if there is overlap.

Further details of determining relative positions of image framesrepresenting a scan of an object are provided in FIGS. 10A-15. FIG. 10Ashows that a first image frame 1002 in a stream of image frames may becaptured as scanning of text document 1000 begins, upon any suitabletrigger.

Next, as shown in FIG. 10B, a succeeding image frame 1004 may becaptured that partially overlaps image frame 1002. In some embodiments,the scanning device may capture the stream of image frames at a ratethat ensures that each new image frame partially overlaps at least oneof the preceding image frames.

As new image frames are being captured as part of the stream of imageframes, a subsequent image frame 1006 that partially overlaps precedingimage frame 1004 may be captured, as shown in FIG. 10C. Further, a newimage frame 1008 may be captured, as illustrated in FIG. 10D. Imageframe 1008 partially overlaps image frame 1006.

Because motion of the smartphone is not constrained, each new imageframe may overlap an immediately preceding image frame as well as otherneighbor preceding frames. As illustrated in the example of FIG. 10D,respective areas of overlap of image frame 1008 with image frames 1002and 1004 are larger than an area where image frame 1008 overlaps withthe immediately preceding image frame 1006. However, in accordance withsome embodiments, each new image frame is, for coarse positioning,positioned relative to an immediately preceding image frame.

FIGS. 11A and 11B illustrate example of a first step that may occur in aprocess of determining a position of a subsequent image frame relativeto a preceding image frame. The first step may be determining an initialestimate of a pose of an image frame with respect a preceding imageframe. In the example shown in FIGS. 11A and 11B, an image frame 1100and next an image frame 1102 may be captured as a user moves thesmartphone over an object to be scanned. In this example, the objectcomprises a text document.

FIG. 11A illustrates initial estimate of a pose of image frame 1102based on information obtained by one or more inertial sensors (e.g.,inertial sensors 206). Initial estimate of pose of image frame 1102 maybe based on a change of output of the inertial sensors between the timesat which image frames 1102 and 1104 are captured. In FIG. 11A, a pose ofimage frame 1100 is schematically shown as (X₀, Y₀, θ₀). In thisexample, X₀ and Y₀ denote a position of image frame 1100 in x and ydimensions, respectively, while θ₀ denotes a rotation of the imageframe. Though not expressly illustrated in FIG. 11 A, a smartphone orother portable electronic device may be oriented in more than just thesedimensions such that more than just these three parameters are used todefine a pose. Separation between the smartphone or other portabledevice acquiring image frames and the object being imaged may alsoimpact the pose such that the parameters defining a pose may include a“Z” dimension represent separation. Tilt of the smartphone in one ormore dimensions relative to the object being imaged may also beparameters that characterize a pose of an image frame. Alternatively oradditionally, characteristics of the image capture may also be regardedas parameters of a pose. For example, the zoom of the camera lens may beregarded as a separate parameter or may be reflected based on its impacton the value of the parameter for the Z dimension. These and otherpossible parameters that characterize the pose are not expresslyillustrated for simplicity.

If image frame 1100 is the first image frame in the stream, its positionmay be taken as an origin for a frame of reference in which other imageframes will be positioned. If image frame 1100 is not the first imageframe in the stream, it may have a position determined relative to apreceding image frame, which in turn may either define the origin orhave a position relative to the origin, through one or more intermediateimage frames. Regardless of how many image frames are in the series,relative image poses of the image frames may define positions for allimage frames.

Regardless of the position in the stream, each succeeding image frameafter the first may be captured and processed as image frame 1102. Aninitial pose of image frame 1102 may be determined with respect to thepose of image frame 1100. During a time between when image frame 1100 iscaptured and when image frame 1102 is captured, the inertial sensorsand/or other sensors indicate a change in the position of the smartphoneor other device by a value of Δx in the x direction and by a value of Δyin the y direction. Also, the sensors used to obtain information on aposition of the smartphone at a time when each image frame is capturedmay indicate a rotation of the device by a value of Δθ. The value ofvalue of Δθ may be determined according to processing as describedbelow. Accordingly, the initial estimate of the pose of image frame 1102with respect to image frame 1100 may be denoted as (X₀+Δx, Y₀+Δy,θ₀+Δθ). Though not expressly shown, changes in other parameters thatcharacterize pose may be similarly determined. For example, changes inorientation or separation in the Z dimension may similarly be reflectedin the new pose estimate.

FIG. 11A illustrates a degree of misalignment between image frames 1102and 1100 that would provide a poor quality image. As shown in thisexample, the respective portions of the text of the scanned object donot match. To align image frame 1102 with the preceding image frame 1100so that a good quality image can be generated, a matching portion of theimage frames may be determined and the image frames may be aligned basedon these portions. In some embodiments, those portions that are within azone of uncertainty are first explored to position image frame 1102 withrespect to image frame 1100. Any suitable technique may be used for thematching, which may be iteratively attempting to find a suitable matchbetween the image frames. FIG. 11B shows image frame 1102 aligned withimage frame 1100 based on the respective content of the image frameswhich is, in this example, the text. The adjusted pose of image frame1102 is shown by way of example only as (X₁, Y₁, θ₁). These values mayrepresent the pose of image frame 1102 relative to the origin of theframe of reference. Though, because these values are derived based onpositioning image frame 1102 relative to image frame 1100, they may beregarded and stored as relative values. It should be appreciated that inembodiments in which more parameters are used to characterize a pose,more than the three parameters illustrated in FIG. 11A would be stored.

Image frames that are locally positioned with respect to preceding imageframes may be stored as a network of image frames, which may then beused for global positioning or other processing. The network maycomprise nodes, representing image frames, and edges, representingrelative position of one node to the next. That network of image framesmay be represented as a graph map or in any other suitable way

FIGS. 12A-D in conjunction with FIGS. 13A-13D illustrate the aboveconcept of building a network of image frames based on local positioningof image frames. A reference point on each image frame, here illustratedas the upper left hand corner of each successive image may be used torepresent the position of the image frame. Relative displacement of thereference point, from image frame to image frame, may be taken as anindication of the relative position of the image frames.

FIG. 13A-D represent respective nodes that may be added to the networkas new image frames are acquired and locally matched with one or moreprevious image frames. Though, in the illustrated embodiment, each newimage frame is matched to its immediately preceding image frame. In thenetwork, any frames that have been locally matched will be representedby an edge between the nodes representing the frames that have beenmatched. Each edge is thus associated with a relative pose of an imageframe with respect to a preceding image frame.

In FIGS. 12A-12C, image frames 1200, 1202 and 1204 are successivelyprocessed. As each new image frame is acquired, its initial poseestimated from navigation information, acquired for example frominertial sensors device, may be adjusted to provide an improved estimateof relative position of the new image frame, by aligning the new imageframe with a preceding image frame. Thus, FIG. 12B shows that, as a newimage frame 1202 is captured, its pose may be determined by matchingimage frame 1202 with a preceding image frame, which is, in thisexample, is image frame 1200. A relative pose of image frame 1202 withrespect to image frame 1200 is thus determined. Similarly, when the nextimage frame 1204 is captured, its relative pose with respect to thepreceding image frame 1202 may be determined in the same fashion, asshown in FIG. 12C.

FIGS. 13A-C conceptually illustrate the building of a network torepresent the matching of successive image frames in a stream todetermine their relative poses. As shown, nodes 1300, 1302 and 1304representing the image frames 1200, 1202 and 1204, respectively, may beadded to the network. In this example, each directed edge schematicallyindicates to which prior image frame relative pose information isavailable for a pair of frames. It should be appreciated that FIGS.13A-13D conceptually represent data that may be stored to represent thenetwork. The network may be stored as digital data in a data structurein computer memory. The data structure may have any suitable format. Forexample, each node may be stored as digital data acting as a pointer toanother location in memory containing bits representing pixel values foran image frame. Other identifying information associated with a node mayalso be stored, such as a sequence number to allow the order in whichimage frames were captured to be determined. Likewise, edges may bestored as digital data representing the nodes that they join and therelative pose between those nodes. Moreover, information relating tocapture conditions, such as a zoom setting or other settings applied tothe hardware that acquired an image frame or status informationgenerated by the controllers for that hardware may be stored inassociation with the nodes. One of skill in the art will appreciate thatany suitable data structure may be used to store the informationdepicted in FIGS. 13A-13D.

As the stream of image frames is acquired, a user may move thesmartphone back and forth across an object to be scanned, possiblytracing over regions of the object that were previously imaged.Accordingly, a new image frame that overlaps multiple preceding imageframes may be captured. In the illustrated example, new image frame 1206that overlaps image frames 1200, 1202 and 1204, as shown in FIG. 12D. Arespective new node 1306 may be added to the network to represent imageframe 1206, as illustrated in FIG. 13D.

In the figures, the dark arrows indicate the relative positionsinitially used to add image frames to the network as part of fastprocessing. The dark arrows also illustrate an order in which imageframes are captured, and the image frames may be said to be “layered” ontop of each other as they are captured, so that the most recentlycaptured image frame is placed, or layered, on top of prior imageframes. Processing that renders a composite image based on theinformation stored in the network may use this overlapping informationany suitable way. In some embodiments, for example, the most recentlyacquired image may be selected or overlapping image frames may beaveraged or otherwise combined to improve the quality or resolution ofthe overall composite image. In other embodiments, processing may selectbetween overlapping image frames to render the composite image based onthe highest quality image frame to render a portion of the compositeimage. In yet further embodiments, when none of the image framesrepresenting a portion of the composite image has suitable quality,processing may generate data to represent that portion of the compositeimage or acquire image data in any other suitable way.

In addition, the possibility of a new image frame overlapping multiplepreceding image frames provides a possibility for a more accuratepositioning of image frames based on global information, meaninginformation other than a match to an immediately preceding image.

Dashed lines shown in FIG. 13D may be a relative position of an imageframe with respect to an overlapping image frame other than animmediately preceding image frame. Thus, node 1306 is shown to beconnected, via respective edges, to nodes 1302 and 1304 which representrespective overlapping neighbor image frames. These edges may be addedas part of processing in the quality track and may be used to morefinely determine positions of image frames, as described in greaterdetail below.

Though FIGS. 12A-12D could be taken as demonstrating a sequence of imageframes as they are captured, they could also be taken as a demonstrationof what could be displayed for a user based on the network being built,as illustrated in FIGS. 13A-13D. As each image frame is captured andlocally positioned, it may be presented on a display device in aposition proportional to its determined position within the compositeimage represented by the network. For example, as the scanning processof the text document begins, image frame 1200 is first displayed. Next,when the user moves the scanning device and image frame 1202 iscaptured, respective larger portion of the composite image of the textdocument may be displayed to the user with a small delay, which may notbe perceived by the user as disrupting or slowing down the scanningprocess. Thus, the composite image on the display may appear to the useras if the object being scanned is being painted on the display as theuser moves the scanning device over the object.

Image stitching techniques in accordance with some embodiments of theinvention may be used to generate a composite image of a scanned objectof any suitable type. As shown in the above examples, the object beingscanned may be a text document, an image, a graph, or any combinationthereof. Further, content the object may be in represented in grayscaleor it may comprise various colors. Image frames representing text, suchas is illustrated in FIGS. 12A-12D, may contain multiple edges or otherfeatures that may be used in aligning image frames. For example, suchfeatures as lines and corners may be used if the scanned object includestext and/or image(s). Though, techniques as described herein are notlimited to such embodiments.

FIGS. 14A-14C show that a relative pose of each new image frame may bedetermined by matching the image frame with a preceding image frame,even if the image does not represent or other content with many featuresthat can be easily identified. To perform the matching, identicalcontent in the matched image frames is determined and may be matchedother than based on corresponding features. For examples regions may bematched based on a pixel-to-pixel comparison, comparisons of gradientsor other image characteristics.

For example, image frames may be aligned using area-based matching. Asshown in image frames illustrated in FIGS. 14A-14C, the content of anobject being scanned (e.g., a photo rather than text) may be an imagehaving content of different color gradient across the image. Hence, thearea-based matching may be suitable for aligning image frames of suchobject. Also, FIGS. 14B and 14C illustrate that motion of a scanningdevice between successive image frames may involve rotation in additionto displacement in an x-y plane. Rotation may be reflected in theangular portion of the relative pose between frames. Though notexpressly illustrated in FIGS. 14A-14C, other parameters, such as tiltand Z dimension also may impact the relative pose.

FIG. 15 is another example of a further technique that may be applied inconstructing a network of image frames as new image frames are capturedand respective nodes representing the frames are added to the network.As in the example of FIGS. 13A-13D, the network is representedgraphically, but in a computer, the network may be represented bydigital values in a computer memory.

FIG. 15 shows the state of the network after a scanning device has beenmoved in one swipe, generally in the direction 1514. In this example,the pose of the first image frame in the network, represented by node1510, may be taken as a reference point. The pose of any other imageframe in the network may be determined by combining the relative posesof all edges in a path through the network from node 1510 to the noderepresenting the image frame. For example, the pose of image frameassociated with node 1512 may be determined be adding the relative posesof all edges in the path between node 1510 and 1512. A pose of eachimage frame, determined in this way, may be used for displaying theimage frame as part of a composite image.

Determining a pose of an image frame based on adding relative posesalong a path through the network also has the effect of accumulatingerrors in determining relative pose of each image frame area alsoaccumulated. Such errors can arise, for example, because of noise in theimage acquisition process that causes features or characteristics in oneimage frame to appear differently in a subsequent image frame.Alternatively, features in consecutive image frames with similarappearances, that actually correspond to different portions of an objectbeing scanned, may be incorrectly deemed to correspond. Thus, for anynumber of reasons, there may be errors in the relative poses. For imageframes along a single swipe, though, these errors in relative pose maybe small enough so as not to be noticeable.

However, as a user swipes a scanning device back and forth across anobject, motion of the scanning device in direction 1524 will generateimage frames acquired at a later time adjacent image frames acquired atan earlier time. In particular, as the path through the network proceedsbeyond node 1512 along segment 1516, eventually, a node 1518 on the pathwill have a position near node 1520. When this occurs, the accumulatederrors in relative positions along the path, including segment 1516, maybe substantial enough to create a noticeable effect in a composite imageincluding image frames associated with nodes 1518 and 1520, if bothnodes are positioned based on accumulated relative poses in paths fromnode 1510. Positioning of image frames in the composite image, forexample, may create a jagged or blurred appearance in the compositeimage.

To provide an image of suitable quality, further processing may beperformed on the network. This processing may be performed in a separate“track” from the processing that is integrating each new image frame inthe sequence into the network. This “quality track” processing may beperformed in a separate process or, in a separate processing thread,than processing to incorporate image frames into the network. In someembodiments, this quality track processing may be performed concurrentlywith processing to incorporate new image frames into the network.However, the specific implementation of the quality track processing isnot a limitation of the invention.

This processing may adjust the relative pose information along the edgesof the network to avoid the effects of accumulated errors in relativepose. Accordingly, during the scanning process in accordance with someembodiments of the invention, as new image frames are being captured andstitched into the composite image, a fine adjustment may be made to thedetermined relative positions of image frames already in the network.Fine adjustments may be made in parallel to the coarse positioning ofsuccessive image frames such that displayed image quality may improve asthe scan progresses. Fine adjustments may be based on global positioningof image frames which may involve determining a position of an imageframe within the composite image based on positioning of image framesother than the immediately preceding image frame.

Other processing techniques may be applied to the composite image as itis being formed or in processing after the composite image is formed.These processing techniques may be based on physical characteristics ofthe image, such as contrast or white balance. Alternatively oradditionally, processing techniques may be based on the content of theimage acquired. An example of a processing technique based on imagecontent is an item removal process.

A piece of paper or other object imaged with a smartphone is often helddown by a user or fixed with the hand for easier capture. As a result,one or more image frames may include distracting features, such as animage of the user's finger or other extraneous items, which may beundesirable to be included in the image of the document.

Accordingly, in some embodiments, processing of image frames to form acomposite image may entail improving quality of the composite image bydetermining a segment of the composite image depicting the user's fingeror other item that is unlikely to be desired in the composite image.Further processing may replace that segment with a less objectionablesegment, such as a segment of a background color or other content of thedetected object.

FIG. 16 illustrates a process 1600 of improving image quality by digitremoval, in accordance with some embodiments. Process 1600 may beperformed without requiring any user input, with improves the overalluser's experience when using a smartphone to capture images ofdocuments.

Process 1600 may begin when capturing of a stream of image frames of anobject by a smartphone is started, at block 1602. This may occur, forexample, when the smartphone operates in an image capturing mode. Theobject may be, for example, a document or any other type of objectincluding content, which may include a text, graphics, and/or images. Inthis example, the object is a page of a document.

An image frame may be acquired at block 1604 as part of the scanningprocess. Next, at block 1606, a focus of a camera of the smartphone anda position of the smartphone relative to the object may be determined inany suitable manner. A shape of the page may be determined in a suitablemanner, at block 1608. For example, the shape of the page may bedetermined based on user input signifying the shape. This input may beprovided in advance, such as by the user selecting a page size ororientation from a menu or other suitable user interface. Alternatively,the page shape may be identified by user input after image capture, by auser providing input indicating the location of the page edges in acaptured image. Alternatively or additionally, the page shape may bedetermined automatically as a result of processing on the image. Edgedetection algorithms may be used. Other information derived from theimage, such as the size and shape of identified lines of text in theimage, may further be used to determine page shape.

Next, at block 1610, a layout of the content of the document may bedetermined. In this step, areas of the document that do not contain, orare unlikely to contain, an image of an undesired item may beidentified. For example, this step may include determining positions oftext, image and table portions of the content of the document.

A representation of a digit (finger) may be detected within the contentof the document, at block 1612. Such processing may apply one or morelinks to identify portions of image containing a digit or otherundesired item. Color, shape, location, presence/absence of shadowsand/or other characteristics may be used to identify a digit within animage.

For example, an application performing digit removal processing may beprogrammed with a palette of colors representing flesh tones such thatdetection of a finger may be performed by identifying regions ofcontiguous pixels in the composite image of a color or colors within thepalette. That palette of colors may be preprogrammed. Alternatively oradditionally, the color palette may be customized for a user during atraining phase in which a user takes a picture of the user's hand. Asanother approach, the palette may be determined adaptively, as the userprovides feedback on whether items flagged by automated processing as adigit are in fact a digit to be removed from the composite image.

Likewise, information about the shape of a digit may be preprogrammed inan application that performs digit removal processing or may be obtainedby user input. Shape may be determined in any suitable way, including byusing known shape recognition algorithms to identify shapes representingan item in an image.

Location of an identified shape may also be used as a factor inprocessing to determine whether the shape represents a digit to beremoved from a composite image. A shape near the edge of a sheet ofpaper may be assigned a higher probability of being a digit to beremoved. Conversely, a shape identified within a portion of a documentdeemed, as a result of processing at block 1610 or otherwise, torepresent content in the image may be assigned a lower left of the claimof being a digit to be removed.

Presence or absence of shadows may similarly be used as an indicator ofa higher or lower probability of a shape being a digit to be removed. Adigit, which is three dimensional, is likely to create a shadow, whichwill appear adjacent the digit in the image. Therefore, detection of ashape having characteristics of a shadow adjacent the shape havingcharacteristics of a digit will increase the likelihood assigned to thatshape being a digit for removal. Conversely, absence of the shadow maydecrease the likelihood assigned to that shape.

Regardless of the number and type of characteristics analyzed to assigna likelihood, the likelihood assigned to a shape being a digit forremoval may be compared to a threshold. If the likelihood is above athreshold, processing at block 1612 may indicate that a digit has beendetected in the composite image.

Similar processing may be performed for removing artifacts representingother undesired items. For example, an application may be programmed toidentify the end of the pen or pencil based on the characteristicsdescribed above or any other suitable characteristics. Accordingly,while FIG. 16 describes processing to remove a digit, it should beappreciated that processing to remove other items may similarly beperformed.

An example of an image of an object including the user's finger that canbe analyzed as described in FIG. 16 is shown in connection with FIGS.17A-17C. In this example, the object comprises a page of a document.

FIG. 17A illustrates an example of an image of a document 1700 processedto remove the depicted user's finger 1702. As shown in FIG. 17A, aborder 1703 of document 1700 is detected, which does not include thesegment representing the finger. FIG. 17A also shows a result of thedetection of the layout of the content of document 1700 (at block 1610of process 1600)—different portions of the content are identified asseparate portions 1704, 1706, 1708 and other areas that are not labeledfor the sake of simplicity.

In some embodiments, the finger representation may be detected using askin modeling-based detecting or any other approach that can be used todetect a representation of a finger. The detection of the finger mayinclude analyzing the distance of the camera to the device to detectpartial representations of the digit in the image of the document, basedon known characteristics of a finger (e.g., a size, shape, texture,etc.).

The next step may include determining, at block 1614, constraints forfilling an area of the image of the document including the finger withother content, without modifying characteristics of the document. Theconstraints may be derived so that the document is cropped and retouchedto remove the representation of the digit without compromising a qualityof a resulting image. Constraints, for example, may limit removal toportions of the image that are outside text areas or other portions ofthe document that may contain useful information that may be degradedthrough attempts to remove the digit. Other constraints may limitremoval to portions where suitable patches may be identified. Forexample, a digit over a region of a relatively uniform color may bereadily patched. However, removing a digit covering a widely varyingpattern may result in a patch and creates image artifacts morenoticeable than leaving the digit in the image. The constraints may bederived based on characteristics of the document, such as its type,size, shape, content, and any other characteristics.

Next, at block 1616, patches of the image of the document depicting theuser's finger may be filled in with other, less distracting, content.For example, the segment representing the finger may be replaced with asegment of a background color or other content of the document, whichmay be performed using any suitable technique. FIG. 17B illustrates theresult of replacing the patch including the finger with the backgroundcolor, as shown in a segment 1710 of document 1700. In this example, thesegment that is replaced is an area inside the page border 1703, andcontent used to replace the segment depicting the finger is thebackground area around the portion of the document including thecontent.

The segment representing the finger in the image of the document may bereplaced with other content in any suitable manner. In some embodiments,a patch match-like approach may be utilized. Such an approach mayentail, for example, identifying a color or colors that blend in withthe colors of surrounding portions to which the patch will be applied.Though, any other techniques may be used.

A patch to remove a portion of an image representing a digit and replaceit with other less visually objectionable information may be applied toany suitable representation of an image. In some embodiments, thepatching technique for digit removal may be applied to a composite imageas the image is rendered. Alternatively or additionally, patching may beperformed on the image as stored within a network as described above.Image frames representing the digit, for example, may be removed fromthe graph map or other network representing a sequence of captured imageframes. These image frames may be simply removed or may be replaced byother image frames synthesized to represent patch. Moreover, it shouldbe appreciated that digit removal as described herein is not limited toapplication the composite images. A single image of a document may bepatched in this way to remove from the image a representation of auser's finger other undesired item.

The image which no longer includes the user's finger may be additionallyprocessed by cropping content of the document and applying otherprocessing, such as performing linear adaptive thresholding, dewarping,and other techniques to improve the quality of the final image shown inFIG. 17C. As illustrated in FIG. 17C, segment 1710 previously depictingthe user's finger is replaced with the segment of the background colorof the document. The image of document 1700 without the distractingrepresentation of the finger may be rendered to the user, at block 1618.

In some embodiments, a portable electronic device (e.g., smartphone 200in FIG. 2) may, at different times, capture image frames at differentresolutions to support, in some instances, faster processing, and inother instances, a higher quality image. In some embodiments, the devicemay automatically switch between modes in which lower resolution imagesand higher resolution images are acquired.

Moreover, in some embodiments, the acquisition of higher resolutionimages may be interleaved with acquisition of lower resolution imagessuch that both types of images may be used to create a composite image.The lower resolution images may be used to create a data structuredefining a framework of the image. Higher resolution images may then beintegrated into this framework. In rendering the composite image, thehigher resolution image frames might be used. Such a framework, forexample, could define a point cloud as described above. The higherresolution image frames could then be positioned relative to the pointcloud by matching features in the higher resolution image frames to thepoints in the point cloud acquired from the lower resolution imageframes.

In the example of a smartphone with a camera capturing a compositeimage, the smartphone may have different operating modes in which imageframes of different resolutions are acquired and processed. The modesmay include a preview mode, a scan mode, and/or modes of other types. Inthe preview mode, the smartphone may acquire a number of image framesthat are used to build a composite image, which may be sometimes serveas a current preview image of a scan. The image frames acquired in thepreview mode are referred to herein by way of example as P-frames.P-frames may be acquired continuously in the preview mode and may beused to provide a continuous real-time view of the camera to a user. Thecamera may operate in the preview mode before the scan mode starts. Forexample, in the preview mode, the camera may operate to continuouslycapture image frames at a frame rate of greater than 15 frames persecond (fps). Though, it should be appreciated that other frame ratesmay be substituted.

A stream of image frames captured when the smartphone is used to scan anobject and used to build a composite image may include P-frames andframes of another type, referred to herein by way of example asK-frames. The P-frames may be captured (e.g., in a preview mode) at auniform rate, for example, under control of the camera's videoacquisition circuitry. The capture of P-frames may be continuous, exceptfor interruptions to use the camera to capture other types of images.For example, K-frames may be captured upon a specific trigger, whichneed not be periodic. The capture of a K-frame may interrupt thecontinuous capture of P-frames. As a specific example, capture of aK-frame may be triggered when an overlap between a K-frame and one ormore of the prior K-frames is smaller than 10%. Using such a trigger mayensure that an entire object being scanned is depicted in at least oneK-frame. Such an approach enables a high resolution composite image ofthe object to be formed from the K-frames. It should be appreciated,however, that any other threshold values may be used to determinewhether to capture a K-frame.

In some devices, a camera will be able to capture only one image frameat a time such that capturing a K-frame may necessitate pausing thecapture of P-frames. The preview mode may therefore be suspended while aK-frame is being captured. Once a K-frame is captured, the preview modemay be restarted. In some embodiments, each time a K-frame is captured,a respective notification may be provided to a user, which may be in avisual or audio format, or a combination thereof.

FIG. 18A shows an example of a stream of image frames 1800 captured by asmartphone camera as object in being scanned. In this example, imageframes 1800 include K-frames 1802A, 1802B and 1802 C, and P-frames1804A, 1804B, 1804C and 1804D. As shown in FIG. 18A, K-frames andP-frames alternate in the stream, and more than one P-frame can becaptured consecutively—e.g., image frames 1804A and 1804B in thisexample.

It should be appreciated that any suitable resolution may be used forK-frames and P-frames. K-frames, for example, may be captured at fullresolution of the camera, or some percentage that approximates fullresolution, such as at least 90% of the available resolution. P-framesmay be captured at a fraction of the resolution of the K-frames.P-frames, for example, may have a resolution of 50% or less of theresolution of the K-frame. In other embodiments, this fraction, forexample, may be less than 40% or less than 30%.

For example, in some embodiments, P-frames may be captured at aresolution of 960×720 pixels (px) or 1280×720 px. In some embodiments,all P-frames in a stream of image frames in a preview mode may haveidentical resolution, as may all K-frames.

The benefit of K-frames may depend on factors such as what is beingimaged and how the camera is positioned relative to the object beingimaged. When a smartphone is being used to scan a document, for example,a higher resolution composite image may be desired to support OCR orother image analysis techniques. Thus, capture of K-frames may beperformed in this mode.

On the other hand, when the camera is held sufficiently close to theobject being scanned, regardless of the numbers of pixels in the imageframe acquired, each pixel may represent a suitably small region of theobject to provide adequate resolution for the contemplated use of theimage being acquired. Accordingly, in some embodiments, a portableelectronic device may be configured to conditionally collect K-framesbased on capture conditions, which may be determined in any suitableway. In some embodiments, a distance to an object being imaged may bedetermined based on camera hardware settings, including a focus distancesetting. Thus, a currently set focus distance and/or other cameraparameters that may be indicative of whether P-frames have adequateresolution for ongoing processing may be used to determine whetherK-frames are to be acquired. In some embodiments, for example, K-framesmay be acquired at distances that provide 280 pixels per inch (ppi) orhigher. Such resolutions may be computed, for example, from the focaldistance from the object, other camera characteristics that indicate thefield of view at that distance, and the total number of pixels in animage array in the camera. It should be appreciated, though, that 280ppt is an example and any other suitable threshold may be used todetermine whether K-frames are captured.

Capturing a stream of image frames including frames of differentresolution, such as P-frames and K-frames, allows displaying the imageframes as a live view. In some embodiments, the P-frames and K-framesmay be processed in separate physical processors or within separateprocesses. Regardless of how processed, the lower resolution of P-framesenables them to be processed more quickly than K-frames. As a result, acomposite image constructed with P-frames may be captured more quicklythan an image with K-frames.

FIG. 19 illustrates a process 1900 of capturing K-frames and P-frames aspart of one or more streams of image frames captured as an object isbeing scanned by a camera of a portable electronic device, which in thisexample may be a smartphone.

Process 1900 may start at any suitable time. For example, process 1900may start when a smartphone is instructed to operate in an imagecapturing mode, which may be done in any suitable manner (e.g., upon auser input). As shown at block 1902, process 1900 begins with thesmartphone in a preview operating mode. In this mode, P-frames may becaptured and integrated into a composite image quickly enough to rendera display of the object in real-time. In this context, real-time meansquickly enough that a user of the smartphone perceives the imageappearing as the smartphone is being moved.

In the preview mode, a P-frame may be then captured, at block 1904.Next, at block 1906, motion sensor data may be acquired from output ofsensors of the smartphone (e.g., inertial sensors 206 in FIG. 2). Itshould be appreciated that the motion sensor data may be received priorto, simultaneously with or after the P-frame is captured, and processingat block 1906 is shown to follow processing at block 1904 by way ofexample only. The motion sensor data may be associated with the P-frameand the information may be stored in memory of the smartphone. Suchmotion information may be used, initially, to integrate the P-frame intoa graph map or other data structure representing a composite image.

Next, at block 1908, the P-frame captured at block 1904 may be insertedinto the composite image (graph-map), which may be performed asdescribed above (e.g., in connection with block 504 in FIG. 5). Next, atblock 1910, image capture conditions may be determined to ascertainwhether the P-frames have adequate resolution to provide a finalcomposite image meeting a resolution criteria, which may be a defaultsetting within the smartphone, may be specified by programming in anapplication that uses the composite image or determined in any othersuitable way.

In this example, the camera capture conditions include a focus distance.A focus distance to the object being scanned may be determined. It maythen be determined, at block 1912, whether the scanning of the objectcontinues or, in other words, whether the “capture loop” is active. Thismay involve, for example, determining whether an indicator to stop thescan has been received. Such an indicator may be based on user input,but may also be generated automatically, such as by tracking a timeoutperiod from the start of preview mode or by automated processing thatindicates either that an image has been acquired or that conditions areprecluding capture of an image with a desired quality.

If it is determined that the scanning of the object is terminated,process 1900 may stop. Alternatively, if it is determined that thescanning of the object continues, process 1900 may follow to decisionblock 1914 where it may be determined whether the captured P-frame has atarget resolution. In the example given above, the target resolution was280 ppi. However, any suitable target resolution may be used. If it isdetermined that the target resolution has not been achieved, process1900 may return to block 1904 where another P-frame may be captured.

If it is determined that the target resolution has been achieved,process 1900 may branch to decision block 1916 where it may bedetermined whether a trigger to capture a K-frame has been detected. Anysuitable trigger conditions may be used, including passage of time,amount of motion of the portable electronic device or overlap of acurrent P-frame with a prior K-frame. If no such trigger has beendetected, process 1900 may loop back to block 1904 where a next P-framemay be captured. Alternatively, if it is determined that a trigger tocapture a K-frame has been detected, process 1900 may follow to block1918 where the preview mode may be stopped. After that, at block 1920, aK-frame may be captured. It should be appreciated that processing atblock 1920 may precede processing at block 1918, as embodiments are notlimited in this respect.

The captured K-frame may then at block 1922 be inserted into a graph mapor other data structure organizing image frames from which a compositeimage may be rendered. The graph map may be the same graph map thatorganizes the P-frames used to generate the preview image. Regardless ofhow the K-frames and P-frames are organized, the preview image maynonetheless be rendered based on only P-frames, notwithstanding theavailability of K-frames. Alternatively, the K-frames may be stored in aseparate graph map and may or, in some embodiments, may not be used inrendering the preview image. In some embodiments, composite images arerendered only with image frames of the same resolution—meaning that apreview image may be rendered using only P-frames and a higherresolution composite image may be rendered only with K-frames.

The preview mode may then be restarted, at block 1924, and a nextP-frame may be captured in the preview mode, at block 1904 as shown inFIG. 1900.

In the described embodiments, an image frame from a stream of imageframes captured when an object is being scanned is processed andinserted into a composite image. The stream of image frames may compriseP-frames interspersed with K-frames. As the composite image is beingbuilt, its quality may be determined and improved. To display thecomposite image to the user in real-time, it may not be practical toprocess all image frames that are captured or to process all imageframes if captured at the full resolution available from the camera.Accordingly, only a portion of the image frames captured, which in someembodiments may correspond to lower resolution P-frames, may be used toconstruct a composite image. Such representative image frames may form apreview image. A preview image may represent a current result ofcapturing image frames as the composite image is being built. Areference image may define the coordinate system and orientation of thecomposite image and may serve as a basis for positioning additionalimage frames into a higher resolution final composite image. Theadditional image frames may be higher resolution K-frames. The K-framesmay be incorporated into a graph map at block 1922. The K-frames may beinserted into a graph map for the composite image that is separate fromthe graph map used to represent the preview image. In this scenario,initial positioning in a K-frame graph map may be determined bycomparison of features in K-frames to features in P-frames alreadypositioned in the P-frame graph map. Alternatively, that graph map maybe the same graph map as is used at block 1908 for P-frames.

FIG. 20 illustrates a process 2000 of updating a reference image, inaccordance with some embodiments. In the illustrated embodiment, thereference image is formed from P-frames. K-frames may also beincorporated into the reference image. Initial position estimates of aK-frame may be based on comparison of features in the K-frame tofeatures in frames already incorporated into a reference image, in thesame way that P-frames are incorporated. The reference image may beprocessed as described above, with such coarse alignment being adjustedby global alignment of image frames, whether P-frames and/or K-frames.

The reference image may be stored in computer memory in any suitableway. In some embodiments, the reference image, containing K-frames inaddition to the P-frames that form the preview image may use the samecomputer memory storage locations that are used to store that data thatis used in rendering the preview image and/or a higher resolutioncomposite image. The preview image, for example, may be rendered byprocessing just the P-frames. A higher resolution composite image may berendered by processing just the K-frames. However, the specific storagearchitecture is not critical to the invention, and any suitable storagearchitecture may be used.

Process 2000 may start, for example, when a smartphone operated to scana scene captures an image frame comprising an image of the scene. Theprocessing depicted in FIG. 20 may be initiated by user input or in anyother suitable way. At block 2002, an image frame may be captured. Next,at decision bock 2004, it may be determined whether the image frame isto be discarded. The image frame may be discarded when its quality isdetermined to be unacceptable for the image frame to be included in acomposite image. For example, the image frame may be discarded when anamount of movement of the image frame relative to a prior image frame isabove a certain threshold value (e.g., in some embodiments, greater than50% relative to a size of a prior image frame), which indicates anexcessive amount of movement. As another example, the image frame may bediscarded when it adds insufficient information to warrant processing.For example, an amount of movement of the image frame relative to aprior image frame is below a threshold value (e.g., in some embodiments,less than 2%), which may be taken as an indication that the smartphonewas not moved from a time when the prior image frame was captured and atime when the succeeding image frame was captured, may be used todetermine whether to discard an image frame. An image frame may also bediscarded if a sharpness of an image represented in the image frame isbelow a certain value, if a shutter speed setting of a camera is slowand excessive motion blur may thus occur, etc. Quality of an image framemay be determined in any suitable way, including using any of thetechniques described herein singly or in any suitable combination. Itshould be appreciated that embodiments are not limited to any specificcharacteristics of an image frame which may render it unacceptable forfurther processing.

If it is determined, at decision block 2004, that the image frame is tobe discarded, process 2000 may return to block 2002 where another imageframe is captured. However, if it is determined that the image frame isnot to be discarded, process 2000 may follow to block 2006 where theimage frame may be pre-processed, which may be performed similar toprocessing at block 402 in FIG. 4. Next, the image frame may beincorporated into a representation of the composite image, at block2008, which may be performed similarly to processing at block 404 inFIG. 4. Like process 400 and other processes described herein (e.g.,processes 500 and 600), process 2000 may be performed under control of aprocessor of the smartphone executing stored computer-executableinstructions or using other suitable circuitry within a smartphone, orin any other suitable way. This processing, for example, may entailincorporating the image frame into a graph map or other suitable datastructure organizing frames to be rendered as the composite image. Inthis context, the composite image may be any suitable composite image,such as the preview image, the reference image or a higher resolutioncomposite image.

In some embodiments, if the image frame is a higher resolution K-frameand the composite image has lower resolution, incorporating the K-frameinto the composite image may entail down-sampling the K-frame to obtainan image frame of a lower resolution (e.g., the same resolution as aP-frame). Such processing may ensure that a preview image is generatedfrom image frames of the same resolution. However, in other embodiments,the same result may be achieved by forgoing incorporation of higherresolution K-frames into the composite image. In yet other embodiments,frames with mixed resolutions may be incorporated into the compositeimage.

After the image frame is inserted into the composite image, at decisionblock 2010, process 2000 may determine whether the image frame capturedat block 2002 has information that is not currently in the referenceimage. An image frame captured as the smartphone is being used tocapture images of an object may be determined to have information thatis not currently in the reference image based on any suitable criteria,including a check of characteristics of the image frame, as describedabove.

If it is determined, at block 2010, that the image frame has informationthat is not currently in the reference image, process 2000 may branch toblock 2012 to update a reference image based on the image frame. Thisupdate may enlarge the reference image using the image frame. As notedabove, the image frame may be stored as part of any suitable datastructure, which may be the same or separate from a data structure usedto hold the K-frames and/or P-frames used in forming a composite image.If the image frame is a K-frame, the reference image may be updated bydownsampling the K-frame to a P-frame. Alternatively, the referenceimage may be updated by a prior P-frame.

Process 2000 may then follow to block 2014, as shown in FIG. 20. If itis determined, at block 2010, that the image frame does not haveinformation that is not currently in the reference image, in thisexample, the processing performed at blocks 2006 and 2008 may representthe processing used to incorporate the image frame into the relevantcomposite image. Accordingly, processing of the image frame may becomplete.

If it is determined that the image frame does not have information thatis not currently in the reference image, process 2000 may continue toblock 2014, where it is determined whether more image frames are to becaptured. For example, as the smartphone is used to scan an object,multiple images of different portions of the object may be captureduntil a user provides input representing a stop command. Accordingly,process 2000 may execute continuously as new image frames are added tothe composite image. While the scan continues, a next image frame may beacquired and process 2000 may loop back to block 2002. Each new framemay be used to expand the extent of the object represented in thecomposite image. If the image acquisition is completed, process 2000 mayend.

In some embodiments, after the reference image is updated, this imagemay be processed, which may involve analyzing quality of content of theimage and replacing a poor quality segment of the image with a segmentof a better quality, thereby improving the overall quality of thecomposite image that can be generated from the reference image. Though,such quality analysis may be performed at any suitable time, including,for some techniques, on image frames before they are incorporated into adata structure from which a composite image is rendered.

It should be appreciated that “replacing” one portion of a compositeimage with another may be performed in any suitable way, and thespecific technique used may depend on the way in which the image framesthat are used to form the composite image are stored and processed torender the composite image. In some embodiments, replacing one portionof the composite image for another may involve altering a data structurestoring the image frames to delete the portions being replaced. Though,in some embodiments, replacing may occur when the image frames areprocessed to render the composite image. The replaced portions may bemarked in some way so that they are passed over during this processing.Alternatively or additionally, the portions replacing other portions maybe marked in some way so that they are processed after the portionsbeing overwritten.

Accordingly, the techniques described herein include techniques foridentifying segments of an image including artifacts arising from arelative position of a source of illumination and a camera. Examples ofsuch artifacts are reflections and/or shadows. Techniques describedherein may be used in identifying other segments that may be suitablereplacements. In this way, segments including reflections and/or shadowsmay be identified and replaced by other image segments, depicting thesame portions of the object being imaged with a higher quality.

In some embodiments, reflections and shadows may be detected byanalyzing at least a portion of the image frames in the composite image.Reflections in an image frame may be formed under any of a number ofcircumstances, such as when a surface of an object being scanned isreflective—e.g., when the object is a glossy reflective sheet of paper,when the object is located behind semi-transparent or translucentmaterial (e.g., glass), and based on any other factors. Reflections thatmay be detected and corrected may be characterized as, for example,self-reflections caused by a flash of the smartphone, externalreflections caused by one or more light sources, and/or externalreflections caused by ambient light (e.g., lighting in an environmentwhere an object is being scanned).

In some embodiments, one or more reflections in the image frame maycomprise one or more specular highlights. A specular highlight may bedefined as a bright spot on an object caused by illumination from alight source. A position of the specular highlight may change with achange in a view direction and as the direction of the light changes.

In some embodiments, a motion of a smartphone from a position when aprior image frame was captured to a position with a currently capturedimage frame was captured may be used to determine whether a detectedspecular highlight is a true reflection caused by a light source or partof intrinsic content of the object being scanned. If a true reflectionis detected, a corrective action may be taken. The corrective action mayrelate to replacing one portion of the image with another. Alternativelyor additionally, the corrective action may entail changing image captureconditions for subsequent image frames, which may entail turning on oroff a flash for the camera. Alternatively or additionally, thecorrective action may entail providing feedback to a user, which mayresult in the artifact not appearing in subsequent image frames.

For example the artifact may be removed by guiding a user of thesmartphone to move the smartphone in a way that allows capturingsubsequent image frames in which the reflection is located in differentpositions. One or more overlapping portions of the image frames acquiredin this way may allow improving the quality of the composite image toeliminate the refection.

FIG. 21 illustrates generally a process 2100 of improving image qualityby removing a reflection. In some embodiments, an image frame from whichthe reflection is removed may be a P-frame. Though, image processing todetect and correct reflection may be performed on K-frames in somescenarios. In some embodiments, the processing may be performed onframes of the type that are being used to form the composite. Moreover,it is not a requirement that the processing be performed on frames, perse. The processing may be performed on different portions of ancomposite image, acquired with the camera that acquired them indifferent positions.

Process 2100 may start, for example, when an image frame is captured andprocessed, at block 2102. In some embodiments, process 2100 may startwhen a reference image is updated—e.g., as shown in block 2012 in FIG.20. To generate a display of a stream of image frames captured as anobject is being scanned with a smartphone in real-time, process 2100 maybe performed on a P-frame, which may be a P-frame captured immediatelyprior to capturing the image frame used to update the reference image.

Regardless of whether the image frame is a P-frame or a K-frame, theimage frame may be first processed as described, for example, at block502 in FIG. 5. Further, the image frame may be inserted into arepresentation of the composite image, at block 2104, which may beperformed in the same manner as processing at block 504 in FIG. 5.

Next, at block 2106, quality of a depiction of a scene in therepresentation of the composite image may be determined. This analysismay involve analyzing the content in one or more portions of thecomposite image for features representative of a reflection. This may beperformed using any suitable technique, including techniques thatidentify regions of approximately the same color and/or brightness in animage. Regions that have a brightness exceeding the brightness ofadjacent regions by a threshold amount may be further processed as aregion likely representing a reflection. Analogous processing, directedat identifying a region having color and brightness characteristics of ashadow may be identified when an analogous process for removing shadowsis performed.

Accordingly, in the example of FIG. 21, at decision block 2108, it maybe determined whether one or more specular highlights are detected inthe composite image, which may be performed using any suitabletechnique. For example, in some embodiments, a specular highlightmodeling approach may be utilized, as is known in the art.

If it is determined the specular highlight is detected, process 2100 mayfollow to block 2110 where it may be determined whether the detectedspecular highlight is an actual reflection or an intrinsic feature ofthe object being scanned. For example, a current image frame may becompared to one or more prior image frame, where each frame isassociated with motion information, to determine whether the specularhighlight “moves” with movements of the smartphone. A further indicationthat a specular highlight is an artifact rather than a representation ofan actual portion of an object being imaged, may be that one or morecharacteristics of the specular highlight (e.g., a size and/or shape)changes from image frame to image frame.

As shown in FIG. 21, when no specular highlight is detected, process2100 may branch to block 2114 where it may be determined whether furtherimage frames are to be captured. Similarly, if the specular highlight isdetected but it is determined to be part of an object being scanned,process 2100 may similarly branch to block 2114.

FIG. 21 shows that, if it is determined that the detected specularhighlight is a reflection in the composite image that should becorrected, process 2100 may continue to block 2112 where feedback may begenerated that can help correct the reflection. The feedback may triggerany suitable corrective action, which may entail adjusting the compositeimage, changing capture conditions or instructing a user how to positionor move the portable electronic device containing the camera.

As shown in FIG. 21, when no specular highlight is detected, process2100 may branch to block 2114 where it may be determined whether furtherimage frames are to be captured. Similarly, if the specular highlight isdetected but it is determined to be part of an object being scanned,process 2100 may similarly branch to block 2114.

If it is determined that the detected specular highlight is an actualreflection in the composite image that should be corrected, process 2100may continue to block 2112 where feedback may be generated forcorrecting the reflection. In this example, generating feedback at block2112 may involve computing a position parameter of the smartphone andgenerating feedback to adjust positioning of the smartphone based on thecomputed position parameter, as described in connection with blocks 508and 510, respectively, in FIG. 5. The smartphone may be moved in anunrestricted manner in three dimensions and can be used to captureimages of an object at different distances and different angles relativeto a plane of the object. Thus, the computed position parameter maycomprise at least one position parameter that does not define a locationwithin a plane parallel to a plane of the scene, at least one positionparameter that defines a spacing between the scene and the smartphone,at least one position parameter that defines an angle of the smartphonewith respect to a normal to the scene, and/or other type of parameter,such as a parameter indicating where to position the smartphone toacquire subsequent image frames.

The feedback may be provided in any suitable form and may include anysuitable information. For example, a visual indicator may be provided ona display of the smartphone indicating to the user a direction in whichto move the smartphone so that the reflection is removed as subsequentimage frames are captured. Additionally or alternatively, feedback maybe output to the user to alter an orientation of the smartphone to avoidreflections or other image artifacts (e.g., shadows).

Next, process 2100 may continue to block 2114 where it may be determinedwhether there are more image frames to be captured, which may be thecase when the scan of the scene is not yet completed and the usercontinues to operate the smartphone to capture images. While the scancontinues, a next image frame may be acquired and process 2100 may loopback to block 2102. If the image acquisition is completed (e.g., if userinput was detected instructing the smartphone to stop the scan), process2100 may end.

When a reflection is detected in the image frame and the user of thesmartphone is guided to change the orientation of the smartphone toacquire subsequent image frames in a manner that can remove thereflection from the composite image, segments in one or more of thesubsequent image frames may be identified that may replace the portionof the image frame where the reflection was detected. The feedback tothe user with respect to acquisition of a subsequent image frame thatwould eliminate a reflection may be, for example, a suggestion of a nextK-frame.

Though, it should be appreciated, that, in some scenarios, no feedbackmay be provided and the user may visually appreciate that, as he or shemoves the smartphone in different orientations with respect to theobject being scanned, the reflection is removed from portions of theimage displayed on the smartphone. Thus, the user may manipulate thesmartphone to capture image frames free of the image artifact. Automatedprocessing may remove the reflection from the composite image, such asby replacing image frames containing the artifact with subsequentlycaptured image frames that do not include it or altering the imagecapture conditions for subsequent image frames.

Thus the artifact may be removed without any specific cues besides achanging location of the reflection in response to a change in theorientation of the smartphone. To aid the user in this regard, when anartifact is detected in a composite image, processing of the compositeimage may include marking the composite image as displayed to the userto call attention to the artifact. As a specific example, the artifactmay be marked with a yellow or red color, which might prompt the user tore-image the portion of the object so highlighted. The additional imageframes captured in this way may be captured from a different orientationand may not include the artifact. Processing to form the composite imagemay replace the image frames with the artifact with subsequentlycaptured image frames, using techniques as described herein.

FIGS. 22A-22D illustrate schematically an example 2200 where asmartphone 102 operated by user 104 may be moved without restriction inthree dimensions to scan an object 2202 in a manner that allows fordetecting and/or removing reflections. In this example, object 2202 isplaced on a support surface 2204. It should be appreciated, however,that embodiments described herein are not limited to an object placedonto a surface, and any type of object positioned in any suitable mannermay be scanned. For example, object 2202 may be held by the user or maybe a scene.

In this example, the object is a document, such as a page from a book ora magazine, a business card, a check for deposit in a bank, a purchasereceipt, or any other type of document. The document may be made of orcovered by a material that reflects light, such as glossy paper or anyother type of material.

In this example, as shown in FIG. 22A, a surface of object 2202 may bereflective so that a composite image 2206 of object 2202 built as theobject is being scanned by smartphone 102 may include a reflection 2208.The reflection 2208 may, based on the relative position of thesmartphone 102 and a light source illuminating object 2202, appear inthe lower left portion of the composite image, which is displayed to theuser as shown in FIG. 22A.

As the user moves smartphone 102, the relative position of smartphone102 and the light source may change. As a result, the position of anyreflection with respect to the composite image may change. Such a changeis shown in FIG. 22B, with the reflection 2208 appearing to move to theright in the composite image. Further movement may further move therelative position of reflection 2208, to a position as shown in FIG.22C. Such a movement may be detected by processing image frames afterthey have been positioned in the frame of reference of a compositeimage, allowing comparison of the location of a reflection in a commonframe of reference from frame to frame.

The fact that the reflection appears to “move” may also be used toimprove the quality of the composite image by removing the reflection.The image frames acquired with smartphone 102 in positions asillustrated in FIGS. 22A-22C, may each contain reflection 2208. However,those image frames may each contain a portion of the object that isobscured by reflection 2208 in the other image frames. As a result,smartphone 102 may capture enough image frames that a composite imagemay be constructed without using portions of image frames containing thedetected reflection.

Moreover, the fact that the reflection depends on relative position ofthe smartphone and the light source means that movement of thesmartphone may remove the reflection. A user may be directed to move thesmartphone to allow capture of image frames without the reflection. Whenthis reflection is detected, for example, using the processing at block2110 in FIG. 21, user 104 may be guided to orient smartphone 102 so thatreflection 2208 is “moved away” from image 2206. This may occurgradually, as user 104 moves smartphone in a particular manner, such asby tilting or rotating the smartphone, and portions of subsequentoverlapping image frames may be used to replace segments having thereflection with segments that do not include the reflection.

It should be appreciated that although FIGS. 21 and 22A-22D show how aquality of a composite image may be improved by removing reflections,similar processing may be performed to remove shadows. Thus, if asegment of an image frame is determined to include a shadow, thissegment may be replaced with a segment of a better quality that does notinclude the shadow.

In some embodiments, to display a composite image in real-time, aquality of a composite image is determined by processing selected imageframes, such as one or more image frames in a reference image. Inembodiments in which only a single type of image frame is captured, thecomposite image may be the same as the preview image. However, inembodiments in which image frames of one type are used to form thepreview image and image frames of additional types are used to form ahigher resolution composite image, the composite image may be differentthan the preview image. Moreover, it is not a requirement that thecomposite image actually be rendered and/or displayed. As used herein, a“reference image” may include a data structure holding image frames thatcould be rendered into an image format, which may be further processedand/or displayed. The reference image may represent a current result ofprogress of the scan of an object. To display the image informationcaptured in the reference image, that information may be processed intoa composite image—i.e., a representation of the reference image that isto be rendered and presented to a user on the smartphone display. Thecomposite image may be rendered with any suitable resolution. In someembodiments or in some scenarios, only a portion of the captured imageinformation in the reference image may be used in rendering a compositeimage. In such a scenario, the composite image may be referred to as apreview image. A lower resolution preview image may be displayed duringa scan process, to enable fast feedback to a user. A higher resolution,final composite image may be rendered upon completion of imageacquisition.

In any event, the quality of the composite image formed at the end of ascan may be improved by improving quality of the reference image.Accordingly, once a current reference image is updated, the updatedreference image and/or the composite image may be processed. Theprocessing may comprise analyzing content of the image to identify areasof different quality and performing further processing of the content,such as, for example, optical character recognition (OCR). In someembodiments, the analysis may be performed on the reference image.However, it should be appreciated that the analysis could be performedon any suitable image information, including on a preview image or acomposite image already rendered.

OCR techniques may be applied to segments of the composite image toassess the quality of those segments. If the quality of one or moresegments is determined to be below a certain threshold, those segmentsmay be replaced with one or more segments of a better quality. OCRprocessing may generate a quality metric on segments of an image.

The OCR processing and verification may be performed on segments of theimage of any suitable resolution that are deemed to contain text. Insome embodiments, this processing may be performed on image frames of ahigh resolution, such as K-frames as described herein. Though, OCRand/or other types of processing may be done on P-frames that depict thesame content. Because a P-frame may have a lower resolution that aK-frame, results of P-frames processing may be displayed to a userfaster, thus improving overall user experience.

The composite image may be segmented by processing performed in responseto any suitable trigger. In some embodiments, the segmenting istriggered when the composite image is updated. Segmentation may identifyareas of the composite image that likely contain text. OCR processing,using techniques as are known in the art or any other suitabletechnique, may be performed on each segment.

Some OCR processes, in addition to associating a particular textcharacter with an item in an image, may output an indication ofcloseness, or confidence, of a match between the item and a textcharacter in an alphabet recognized by the OCR process. Such a closenessmay be determined using a correlation operation or any other suitableprocessing.

The closeness indicators may serve as a quality metric for a segment.Using averaging or other suitable process, the closeness indicators forall of the characters identified in a segment may be combined into aquality metric for the segment. As a result, when OCR processing is ableto identify characters in a segment with a high average closeness, thesegment may be assigned a high (“good”) value of the quality metric. Incontrast, segments with low closeness may be assigned a low (“bad”)value of the quality metric.

Specifically, the image may be divided into areas. These areas may haveany suitable shape, but, reflecting the convention that text ispresented in parallel lines, these areas may be boxes. The segmentationprocessing may rely on conventional layout characteristics of documentsto identify segments. As one example, segmentation may identify imageareas of comprising one or more content portions and image areascomprising a peripheral portion. The content portions may be recognizedbased on one or more lines of characters, whereas the peripheralportions may be identified as areas of background color that do notinclude content, for example.

It should be appreciated that embodiments are not limited to anyspecific ways of segmenting the composite image. Because segmentationmay be performed successfully on low resolution images, the segmentationof the composite image may be performed, for example, on P-frames or acomposite image being formed with the P-frames.

FIG. 23 illustrates a process 2300 of segmenting a composite image whenit is determined that it has been updated. Process 2300 may start whenan image frame is captured and pre-processed at block 2302, which may beperformed, for example, similarly to processing at block 502 in FIG. 5.As the next step, the image frame may be incorporated into arepresentation of a composite image, which may be performed as describedherein—e.g., similarly to processing at block 504 in FIG. 5.

Next, it may be determined at decision block 2306 whether a referenceimage has been updated. If this is the case, process 2300 may follow toblock 2308 where content of the composite image may be segmented. Thesegmentation may include identifying areas of similar content, which maybe shaped as “boxes,” which may be processed separately, to improvequality of the composite image.

In some embodiments, results of segmenting the image may be presented onthe display of the composite image in a form of feedback, at block 2310.For example, frames or other visual features, which may have differentcolors, may overlay the image on the display to emphasize detectedsegments. Additionally or alternatively, visual features that indicatequality of the segments may be displayed. As a specific example, linesmay be overlaid on a rendered composite image that is being displayed toindicate the boundaries of the segments. These lines may have colors,such as green, yellow and red, to indicate the quality metric assignedto the corresponding segment. Though, other indicators may be used. Forexample, areas of low quality may be shown “greyed out” or in some othersuitable way to alert the user to the area of low quality.

If it is determined at decision block 2306 that the reference image hasnot been updated, which may occur for example if the captured imageframe is redundant of prior image frames or of too low a quality,respective feedback may be generated at block 2310. This feedback may bebased on a previously prepared segmentation or may be updated to correcta reason that the image frame was not used. For example, if the imageframe was not used because the camera was not moved, the displayedfeedback may prompt the user to move the smartphone. Regardless of thefeedback, process 2300 may then follow to block 2312, where it maydetermine whether there are more image frames to capture. For example,if the scan of the object is still in progress, process 2300 may loopback to block 2302 where another image frame may be captured andprocessed. If it is determined at decision block 2312 that no furtherimage frames are to be captured (e.g., when an indication to stop thescan is received or the scan is stopped in other way(s)), process 2300may stop.

FIG. 24 illustrates in more detail a process 2400 of segmenting contentof an image representing the composite image, in accordance with someembodiments. The image may represent an entire composite image or aportion of the composite image. This processing may be performed on thecomposite image, of any suitable resolution, and/or a reference image.FIGS. 26A-26D show an example of segmentation of such an image 2602(FIG. 26A) which is, in this example, an image of a document includingtext and/or other characters. The segmentation may be followed byfurther analysis of the image content, such as by optical characterrecognition and any other processing.

Process 2400 may first identify areas of low quality, at block 2402.This may include identifying areas including shadows, reflections or anyother artifacts that contribute to the decrease in the image quality.Additionally or alternatively, areas that include no content (e.g.,areas that include only background color) may be identified as lowquality area. Exact boundaries of the identified low quality areas maybe delineated. This delineation may be achieved by storing data incomputer memory that identifies other data representing a particularsegment of the image. Alternatively or additionally, the delineationsmay be presented on the display along with the image.

After the areas of low quality are identified in the composite image,next, at block 2404, a layout of the content in the remaining areas maybe identified. The remaining areas may be analyzed to identify segmentscontaining text. Such processing may be performed by identifying linesof characters, aligned with an edge of a document, or in any othersuitable way. In the example illustrated, these areas may be representedby boxes. FIG. 26B illustrates layout boxes 2604, 2606 and 2608 that areidentified within image 2602 and that bound respective areas of contentof the image. In this example, box 2604 includes a title of thedocument, and boxes 2606 and 2608 each include a respective paragraph inthe text of the document.

It should be appreciated that the layout boxes are shown by way ofillustration only and that, in embodiments where content of thecomposite image is complex, areas of shapes other than boxes may beidentified and processed.

Next, at block 2406, the identified box may be associated with framesforming the composite image. The segments found in 2404 may be added tothe box map of the previous iteration of the process 2400. For example,as a result of process 1900, the reference image may be updated and theeffectively captured area may increase. Thus, new areas (the “remainingareas” in 2402) are found. The box map of the previous iteration may beoptimized by either enlarging existing boxes or adding new boxes.

A further act of process 2400 may include determining a position of anext high resolution image frame to be captured. The position of suchimage frame may be identified so that the frame overlaps a current imageframe and covers a portion of a segment to be further analyzed. In thisexample, determining a position of a next image frame may compriseadjusting the conditions under which a next K-frame is captured.

As shown in FIG. 24, next, feedback may be generated at block 2410. Thefeedback may indicate to a user a manner in which to move the smartphoneto result in a capture of a K-frame in the determined position. Suchfeedback may be presented by displaying the composite image to presentthe layout boxes that were detected and adjusted at blocks 2404 and2406, respectively. An example of such feedback is shown in FIG. 26B,illustrating layout boxes 2604, 2606 and 2608. The layout boxes may becolored, shaded, or presented with any suitable visually distinctivefeatures to identify segments for which further image frames aredesired. Alternatively or additionally, the feedback may be presentedusing visual or audible signals to the user to indicate a direction inwhich to move the smartphone to acquire a desired K-frame.

The bounding boxes may be positioned so that different portions of theentire textual content of the image are included in respective boxes.Each of the bounding boxes 2604, 2606 and 2608 may have a differentcolor. In some scenarios, a color may be used to designate acharacteristic of the bounding box, such as its size, type of content,etc. It should be appreciated that three bounding boxes 2604, 2606 and2608 are shown by way of example only, and any suitable number of boxesor bounding areas of other shapes may be identified.

Next, at block 2412, process 2400 may wait for a next update of thereference image. When such next update occurs, process 2400 may returnto block 2402, as schematically shown in FIG. 24. Alternatively, process2400 may then end.

In some embodiments, determining quality of a segment in a compositeimage comprises determining a relative image quality based on quality ofoptical character recognition. For example, when the object being imagedis a document with characters, the likelihoods or other closenessmetrics associated with identifying characters may indicate imagequality.

Accordingly, after the image representing an object being scanned hasbeen segmented, content of at least a portion of the image may beanalyzed by applying optical character recognition techniques to contentof the image. FIG. 25 illustrates process 2500 of applying opticalcharacter recognition (OCR) techniques to content of the composite imageand assessing quality of the OCR. The OCR quality assessment may beperformed in real-time, as each new box is identified and updated in thesegmented image.

OCR may be performed using any suitable module comprisingcomputer-executable instructions that, when executed, performs an OCRtechnique. In some embodiments, an OCR engine may recognize charactersin multiple languages and may comprise dictionaries for one or morelanguages. The OCR engine may be stored in memory of the smartphone(e.g., as part of applications 222 in memory 220) and may thus be usedwithout requiring the smartphone to be connected to the Internet. TheOCR engine may be received from a third party or may be implemented inany suitable way.

OCR processing may require a certain amount of time, which may take upto a few seconds. Thus, to provide results of the OCR processing withouta delay, the processing may utilize both low resolution frames (e.g.,P-frames) and high resolution frames (K-frames). In particular, the OCRprocessing may be performed so that a result of segmentation of one ormore P-frames may be used in association with performing OCR on aK-frame representing at least a portion of an identified segment, asdescribed in more detail below.

Process 2500 may start when a box is selected within the compositeimage. The box may be selected, for example, from boxes identified atblock 2406 in FIG. 24. The box may be selected at block 2502 in anysuitable manner. For example, when an object being scanned is a documentwith characters, a box in the upper left corner of the document may beselected first. Continuing with the example in FIGS. 26A-26D, box 2604may be selected at block 2502. Though, it should be appreciated that thebox may be selected in other manner. Furthermore, it should beappreciated that processing to achieve image segmentation as shown inFIG. 24 and OCR processing on previously identified segments may beperformed simultaneously (e.g., processes 2400 and 2500 may run inrespective different threads).

Next, at block 2504, an area within the box may be selected in a highresolution frame. For example, when the scanned object is a documentdepicting text, the area may comprise a rectangle encompassing a line ora group of words. Only an area having content that has not yet beenrecognized may be selected. Because OCR of a good quality may require ahigh resolution image, the OCR may be performed on a K-frame. Thus,after the area within the box identified within a P-frame is selected, arespective K-frame may be identified such that the K-frame includes thesame content as content bounded by the box identified within theP-frame. Thus, at block 2504, the area is selected from a frame having ahigh resolution, which is, in this example, a K-frame.

The OCR is then performed on the selected area, at block 2506, which maybe performed using any technique as known in the art. Quality of the OCRprocessing may be verified in any suitable manner and the selected areamay be associated with an indicator of the OCR quality, such as a score(e.g., a confidence score) that indicates a likelihood of a correctrecognition of characters. Feedback may be generated based on results ofthe OCR quality verification, at block 2508. For example, differentvalues or average values of the scores may be assigned to differentquality levels. These quality levels, for example, may represent anacceptable, unacceptable and marginal quality. As shown in FIG. 26D,text within one or more areas of box 2604 may be highlighted to indicatethat the text has been recognized. Different portions of the text may behighlighted with different colors or may otherwise be emphasized toindicate areas that have been recognized and respective quality ofrecognition. For example, one color (e.g., green) may be used toindicate confidence scores meeting the acceptable threshold, whereasanother color (e.g., yellow) may be used to indicate confidence scoresassociate with the marginal level of quality. Any other visualindicators or cues of quality of OCR application may be utilized, asembodiments are not limited in this respect.

In some embodiments, feedback indicating whether one or more areas ofthe composite image may need to be rescanned may be generated. This maybe required and/or recommended to improve the quality of the OCR ofcontent of the composite image. Such an indication may be based on thequality metrics for the segments, such that segments with qualitymetrics below some threshold are indicated for rescanning.

It may then be determined, at decision block 2510, whether the compositeimage includes more boxes to analyze. If it is determined that moreboxes are to be processed, process 2500 may loop back to block 2502where that box may be processed. The next box may be selected in anysuitable way and the results of the OCR processing may be presented tothe user in a manner so that the user perceives a simultaneous analysisof the quality of the application of OCR to the entire document or aportion of the document represented in the composite image.

Finally, if it is determined that there are no more boxes to beprocessed (e.g., when the scanning of the object is completed), process2500 may end.

Other image correction techniques may address image blur. The speed theuser moves the portable electronic device to capture the composite imagemay influence the image quality. For example, if the user moves asmartphone over a large distance during the exposure time for a singleimage frame, then artifacts in the image may appear, such as motionblur. Motion blur includes any artifacts from moving the camera whilethe image is captured. Additionally, motion blur can arise from anobject moving in the field of view of the camera during capture of imageframes used to form a composite image. In some scenarios, it may bedesirable to reduce these artifacts in order to improve the imagequality of the composite image.

An indication of motion blur may be determined in any suitable way. Insome embodiments, motion blur may be detected by analyzing at least aportion of the image frames in the composite image. Any suitableprocessing may be performed on one or more image frames to compute anindication of motion blur. For example, processing techniques mayidentify objects in image frames with diffuse edges. The degree ofdiffusivity may provide an indication of image blur. Alternatively oradditionally, processing techniques may identify objects depicted inimage frames that, from frame to frame, change position relative toother objects in the image frames. The amount of motion, relative to thetime between image frames may provide an indication of motion blur.

Alternatively or additionally, an indication of motion blur may bedetermined based on sensed motion of a portable electronic device beingused to capture image frames. In some embodiments, motion may be sensedusing inertial sensors mounted on the portable electronic device, imageanalysis or any other suitable technique. In some embodiments, focusdistance or other information indicating a distance to objects beingimaged may be used in combination with motion information to determinean amount of relative motion within an image frame of an object beingimaged during the exposure time for capturing that image frame. Acomputed amount of motion during the exposure time may provide anindication of motion blur.

Regardless of how motion blur is detected or indicated, if motion bluris detected, a corrective action may be taken. The corrective action maybe conditionally taken based on the amount of motion blur detected. Forexample, in some embodiments, corrective action may be taken based onthe amount of motion blur exceeding a threshold, which may be determinedin any suitable way. The specific action taken also may be selectedconditionally based on hardware capabilities of the device capturing theimage frames and/or capture conditions.

The corrective action may entail changing image capture conditions forsubsequent image frames, which may entail adjusting one or moreparameters of operation of the camera, such as setting an exposure time,turning on the flash, and/or increasing the camera gain. Alternativelyor additionally, the corrective action may entail providing feedback tothe user, which may result in reduced motion blur in subsequent imageframes. Such feedback may be an indication to the user to slow down themovement of the portable electronic device or to turn on the cameraflash. Examples of such indications are depicted in FIGS. 28A-28B bytext messages 2802 and 2804. This indication may take on a variety offorms and may include, but are not limited to, a textual message, anicon, a light indicator, or a sound message.

Reducing motion blur may occur by determining camera parameters, such asexposure time, gain, flash on/off, based on the current lightingconditions and one or more parameters of movement of the camera. In someembodiments, the camera parameters may be adapted in real-time to reducethe motion blur when capturing a sequence of images to form a compositeimage. Additionally or alternatively, the user may be informed inreal-time how to improve the image capturing process, such as by turningon the flash or by moving the portable electronic device slower.

To reduce motion blur, the exposure time may be set so that movement ofthe portable electronic device or of objects being captured in the imagedoes not create an observable artifact in the image. The degree to whichan artifact is observable may depend on factors such as display size andresolution such that an acceptable amount of motion blur may bedifferent for different embodiments. An acceptable degree of motion blurmay be determined empirically, by computation or in any other suitableway. In some embodiments, an acceptable degree of motion blur may berepresented as a number of pixels of motion of an object within a singleimage frame.

An exposure time corresponding to an acceptable degree of motion blurmay be determined in any suitable way. In some embodiments, an exposuretime may be determined based on the movement speed of the portableelectronic device and a maximum motion blur that is desired. As anexample, if the user moves a smartphone at a speed v during an exposuretime t, then the image will have a motion blur of s pixels. The numberof pixels may be determined based on sensed motion, focus distance,field of view angle, and/or any other suitable parameters that maytranslate, through geometric calculations, a motion of the portableelectronic device to a motion across an image frame of an object in thatscene. The movement speed v, which can include lateral and/or rotationalspeed, of the camera may be determined from the motion sensors in thecamera device.

A threshold value may be set for the motion blur to stay below in orderto provide an acceptable level of motion blur artifacts in the image.This threshold value may be expressed as a distance s, in numbers ofpixels, the camera may move during exposure to provide an acceptablelevel of motion blur. Such a threshold value may be based on user inputand/or predefined operating settings for the device. The threshold valuemay also change during capture of a composite image and/or betweencomposite images and/or may be selected based on an applicationprocessing and image being captured or in any other suitable way. Basedon the threshold for the distance s and the movement speed v, a targetexposure time t may be calculated. Setting the exposure time for alonger amount of time than the target exposure time may result in motionblur above the desired level and poor image quality in the capturedimage.

In some embodiments, the exposure time may be correlated with otherimage capture parameters, and a control algorithm for the portableelectronic device may adjust exposure time in conjunction with theseother parameters. For example, exposure time may be correlated withcamera gain. For a given light condition, camera gain may need toincrease as exposure time decreases in order to provide an image withsufficient image contrast. Accordingly, upon changing the exposure timeto limit the level of motion blur, a control component within a portableelectronic device may alter one or more parameters of camera operationto offset that change.

For example, the camera gain may be increased in order to compensate fora lowered exposure time for the image frame. However, increasing thecamera gain may introduce noise in the captured images. Therefore, itmay be desirable to select the camera parameters for a desired exposuretime and camera gain during image capturing that balances exposure timeand gain based on lighting, degree of motion and/or other captureconditions. In some embodiments, this selection of exposure time andcamera gain occurs in real-time. Some embodiments alternatively oradditionally may include feedback provided to the user as part ofselecting camera parameters.

As an example, based on the current lighting conditions, an exposuretime and gain may be determined that collectively enable the image to beproperly exposed during image capture. However, when the currentsettings for exposure time and gain are not appropriate given currentcapture conditions (such as lighting or camera motion), the current gainand exposure time may then be changed. Parameters of the camera hardwareor other factors may limit the amount to which the gain or otherparameters may be changed. In some embodiments, a threshold gain valuemay limit settings on the current gain value. This threshold gain valuemay be a parameter in the portable electronic device that can bepredefined, specified by an application using the image being captured,and/or based on user input. The threshold gain value may indicate alevel of gain above which the captured images may appear undesirablynoisy.

In some embodiments, operating conditions may be determined dynamicallybased on sensed capture conditions. For example, the exposure time maybe set to the target exposure time, computed based on the current gainsetting and sensed lighting conditions, if the camera hardware supportssuch a setting. If the current gain is below the threshold gain value,the current gain may be adjusted during image capturing to provide alevel of motion blur below a desired level given the current exposuretime. In some embodiments, a control system controlling the portableelectronic device may set the gain and exposure time such that theexposure time is shorter than the threshold exposure time and the gainis lower than the threshold gain, if such an adjustment is possible.

In some embodiments, when adjustments of exposure time and gain (orother parameters of camera operation being adjusted) cannot be set atlevels below the threshold values for these parameters, other correctiveaction may be taken. Such corrective action may include adjusting otherparameters or adjusting capture conditions. For example, a flashassociated with the camera may be activated to increase the amount oflight of the scene being imaged. Alternatively or additionally, feedbackmay be provided to warn the user that conditions may yield anundesirable image and to alter image capturing conditions, such as tomove the portable electronic device more slowly. Additionally oralternatively, the user may be notified to adjust use of the cameraflash to alter the lighting conditions used for determining the targetexposure time and target gain. Feedback to the user to change the flashsettings on/off may be provided, an example of which is provided by 2804in FIG. 28B. Such feedback may be in the form of a text message, anicon, light indicator, or sound message. This feedback indicator maydisappear once the current gain is no longer above the threshold gainvalue.

FIG. 27 illustrates a process 2700 that may be used to improve imagequality by reducing motion blur. Such a process may be executed, forexample, by a processor within a smartphone capturing image frames orother suitable processor. In this example, the process may adjustparameters of camera operation to achieve motion blur below a desiredlevel. The process may be performed iteratively until suitableparameters within any thresholds set for those parameters aredetermined. In this example, the exposure time is set to a level thatachieves an acceptable level of motion blur, if such a setting ispossible given current lighting conditions. If not, camera gain isadjusted, if possible within the threshold gain, and a further attemptis made to set the exposure time to a time that yields motion blur at adesired level while providing adequate exposure time given currentlighting conditions.

In the example of FIG. 27, process 2700 may start, for example, when asmartphone operated to scan a scene captures an image frame comprisingan image of the scene. The processing depicted in FIG. 27 may beinitiated by user input or in any other suitable way. At block 2702, animage frame may be captured. Next, at block 2704, the current cameramotion data based on camera motion sensors (ex. lateral and rotational)is acquired.

From the current motion data obtained at block 2704, a target exposuretime may be determined in block 2706. As an example, the motion data mayinclude the movement speed of the smartphone, and the target exposuretime may be determined based on the movement speed, focus distance to anobject being imaged, and/or other information. The smartphone movementmay be lateral and/or rotational displacements. At decision block 2708,it is determined whether the target exposure time can be set for imagecapture based on current gain and other image capture parameters.

Processing at decision block 2708 may be based on comparing the targetexposure time to operating characteristics of the camera. Alternativelyor additionally, processing at decision block 2708 may be based oncurrent camera conditions, such as lighting. Lighting may be determinedbased on analysis of the image frame obtained at block 2702 or in anyother suitable way. As a specific example, a processor controlling asmartphone may be programmed with a table, equation, or other suitablerepresentation of a relationship between lighting conditions and minimumexposure time required to capture enough light energy to produce animage of acceptable intensity.

If as determined by processing at decision block 2708, it is possible toset the current exposure time to the target exposure time, the exposuretime is set to the target value in block 2710. The processing then endsfor the current image frame, and process 2700 proceeds to decision block2712. If additional image frames are being captured such that thereremain more image frames for processing, processing loops back fromdecision block 2712 to block 2702, where the next image frame isacquired and processing repeats for that frame. Processing continues inthis loop until no more frames remain, at which point process 2700 ends.

Regardless of the criteria used to determine allowable settings forexposure time, if processing at decision block 2708 determines that theexposure time cannot be set to the target value, then the process movesto decision block 2714 where the current gain may be compared to athreshold gain value. The threshold gain value may be an upper limit tokeep image noise at a desired level. That upper limit may be determinedbased on characteristics of the imaging array within the camera beingused to capture image frames, or in any other suitable way. In such anexample, if the current gain is less than the threshold value, then thecurrent gain may be increased in block 2722.

Processing in block 2722 may entail computing a target gain value. Thetarget gain may be determined based on exposure time and lightingconditions. Such a value may be computed based on those parameters andor may be determined based on image characteristics for the image framebeing processed. Accordingly, the target value of gain may be determinedin any suitable way to provide suitable image characteristics. Forexample, the gain may be adjusted to provide average pixel values acrossthe image of one half of the maximum pixel value. As another example,the gain may be adjusted to provide no more than a predetermined numberof pixels in the image frame having a maximum image value. As yetanother example, the gain may be adjusted to provide no more than apredetermined number of pixels in the image frame having a value below athreshold. As yet a further example, multiple criteria may beconcurrently applied in determining the target gain. Once the targetgain is determined, the current gain may be set to the target gain, ifthat target gain can be achieved with the camera hardware being used toacquire image frames. Otherwise, the current gain may be set to themaximum gain for that hardware.

It should be appreciated that, though the example of FIG. 27 shows thatthe gain is increased at block 2722, in some embodiments, data may beincreased or decreased at block 2722. A decrease may occur, for example,when the current gain is greater than the target gain.

As an example of another possible variation, in some embodiments,processing at block 2722 may increase the current gain to the thresholdgain value. However, in the embodiment illustrated, the exposure time isset to the longest suitable time so that, when possible, the camera mayoperate at a gain below the threshold gain value. Operation under thesecharacteristics may provide less overall image noise. Accordingly, atblock 2722, a target gain may be computed and the current gain may beset based on that target gain, regardless of whether setting the gainresults in an increase or a decrease in the current gain.

After the current gain is changed based on the computed target gain, thecurrent gain is compared to the threshold gain value. When the thresholdgain value is an upper limit, decision block 2724 determines if thecurrent gain is lower than the threshold gain value. In someembodiments, the threshold gain value the threshold gain may equal themaximum possible gain for the camera hardware. In other embodiments, thethreshold gain may be set to some fraction of the maximum possible gain,such as 90% of the maximum. In such embodiments, the threshold value mayprevent the current gain from reaching the threshold gain value. In someembodiments, the threshold may be determined based on factors instead ofor in addition to the maximum possible gain supported by the camerahardware. For example, in some embodiments, the threshold gain may becomputed based on current lighting conditions, other camera parameterscapture conditions or settings and/or could also be a user input.Regardless, in some embodiments, the threshold gain may be selected toprovide acceptable image noise under current conditions.

If the current gain is below the threshold gain value, processing mayreturn to block 2712, where additional image frames may be processed.However, if the computed target gain is higher than the threshold gain,signifying that the camera cannot achieve the target gain, thencorrective action, other than automatically changing the camerasettings, may be taken. As an example, the user may be prompted tochange some parameter of operation of the portable electronic deviceused to capture image frames. Such prompting may be delivered as awarning or indicator displayed to the user to slow the smartphonemovement by block 2720. Examples of such an indicator may be a textualmessage (as shown in FIG. 28A by 2802), an icon, a light indicator, or asound message.

Though not illustrated in the example of FIG. 27, a user prompt may besubsequently removed. Processing may determine when to remove the userprompt in any suitable way. In some embodiments, the warning orindicator may be displayed until the user moves the portable electronicdevice at a slower speed. Alternatively or additionally, the prompt maybe removed after the passage of time or detection of any other suitablecondition.

Conversely, if the processing at decision block 2714 determines that thecurrent gain is not less than the threshold gain, then processingproceeds to decision block 2716 where a determination is made whetherturning on a camera flash is possible for the portable electronicdevice. If a flash is available to turn on, then an indicator may beshown to the user by block 2718. This determination, for example, mayentail determining that the flash is not already on and/or determiningthat a flash, if turned on, might provide adequate illumination to allowimage frames meeting quality criteria to be captured at the targetexposure time. Such an indication may be a textual message (as shown by2804 in FIG. 28B), an icon, a light, or a sound message. In someembodiments, the indication may disappear when the flash is on or afterthe passage of time or in response to any other suitable condition, suchas receiving an image frame for processing with illumination sufficientto meet the criteria imposed at decision blocks 2708 and/or 2714.Additionally or alternatively, the flash may be automatically turned onbased on current lighting conditions, image capture conditions of theportable electronic device, and/or user input. If turning on a flash isnot possible, then a warning or indication to the user to slow thesmartphone movement is displayed by block 2720, as described previously.

Processing as described herein may be used to scan a document to acquirein the memory of a portable electronic device, or other suitableprocessing device, a composite image of the document. However,processing as described herein is not limited to scanning of documents.A composite image may be assembled of any scene containing one or moreobjects. In this way, panoramic images may be acquired.

When scanning the scene, image frames are captured and assembled into arepresentation of a composite image using one or more of the techniquesdescribed herein. In addition to storing a representation of thecomposite image in computer memory, the composite image may be renderedon a user interface. That rendering may occur in real time, such thatthe user views the composite image as the scene is being scanned.

Image frames may be added to the representation of the composite imagesuch that the composite image may grow in size as the user scans alarger and larger scene. A portable electronic device capturing imageframes may detect a condition to stop and/or pause the processing of theimage frames that add to the composite image. The portable electronicdevice may stop the sequential processing of the image frames based on adetected stop condition.

A stop condition may be implemented automatically by the portableelectronic device. A stop condition may be implemented by detecting anoutput of a motion sensor on the portable electronic device as thedevice moves and/or tilts. In some embodiments, the stop condition mayoccur when the portable electronic device is no longer positioned tocapture the current scene, such as when the device tilts by more than athreshold angular amount from the orientation used to capture imageframes or moves at a speed that exceeds a rate at which the camera cancapture image frames with motion blur exceeding a threshold.Additionally or alternatively, the user may provide input to implement astop condition, such as a button, an icon, a voice recognition, and/orby manually interacting with the user interface of the portableelectronic device.

A stop condition may also be used when capturing of a composite image iscomplete. Alternatively or additionally, in some embodiments, a user maywish to scan a scene as multiple regions captured at different times orotherwise construct a composite image using image frames captured otherthan as part of a continuous stream of image frames. In that scenario, astop condition may be applied as each region of the composite image iscaptured.

Accordingly, a portable electronic device capturing image frames asdescribed herein may identify a resume condition. A resume condition maybe used to continue processing of the image frames to form the compositeimage. One or more types of processing may be performed in response to aresume condition. In some scenarios, the portable electronic device mayresume processing image frames for addition to a representation of acomposite image in response to detecting the resume condition. As withthe stop condition, the resume condition may take any suitable form,including express user input or an implied user input, such as detectinguser actions signifying that the user is pointing the camera of theportable electronic device again at the scene.

In response to the resume condition, a processor may relocalize theimage frames being captured following the resume condition with respectto the image frames captured prior to the stop condition being detected.Relocalization may be performed using a spatial correlation of an imageframe to the representation of the composite image already stored. Insome embodiments, the spatial correlation of the image frame to thecomposite image includes matching the image frame to an image frame ofthe composite image. Such a technique may enable the portable electronicdevice to continue processing the image frames for the composite imageas if the stop condition has not been implemented. That matching may beperformed using techniques as described above or in any other suitableway. Alternatively or additionally, the positional information may beused to determine a rough spatial correlation with image matching usedto adjust the correlation.

One or more types of continued processing of image frames may beemployed after a resume condition is detected. The specific may dependon the implementation of the portable electronic device performing theprocessing. Alternatively or additionally, a portable electronic devicemay be configured to operate in one of multiple modes in which differentprocessing may be performed following the resume condition. For example,processing following the resume condition may add image frames to thecomposite image, extending the image or overwriting any portion of thecomposite image representing a portion of the scene that is scanned. Asanother example, processing following the resume condition may replaceonly a designated portion of the composite image with new imageinformation from a newly acquired image frame or frames.

As a specific example, after capturing the composite image, the user maywant to change or update a specific area of the composite image. As anexample, if the user is taking an image where a moving car or person iscaptured, then the user can selectively recapture the area of the movingobject to remove it from the image. As another example, selectivelyupdating a specific region of the image may be used for adding an objectinto the image, such as in forming an image of a complete group ofpeople. Instead of missing a first person operating the camera in thecaptured image, the first person can scan a scene, forming a compositeimage of the scene, including the location where the first person shouldappear. The first person may then stop image capture. A second personwho was in the scene when the composite image was canceled may step outof the scene to operate the portable electronic device. The first personmay enter the scene and the second user may designate the region of thecomposite image occupied by the first person for replacement. The secondperson may then provide input acting as a resume condition to theportable electronic device to rescan at least that portion of the scene.The processing on the portable electronic device may place the portionof the composite image now containing the first person with new imageframes, which will represent that portion including the first person. Inthis way, both the first person and the second person will appear in thefinished composite image.

In another example, an object may be captured incorrectly, such asduring scanning. Instead of rescanning the entire scene, the user mayselectively update the appropriate area of the image. Such a techniquemay also be used if there is an error in obtaining the initial compositeimage and a specific region needs to be recaptured.

If this type of dynamic updating of a captured composite image isdesired by the user upon capturing an initial image, the user may selecta region to update or change. Selection may happen manually by touchingthe screen with either the user's hand and/or a tool, such as a styluspen. In some embodiments, areas may be automatically selected byprocessing that identifies a moving object across multiple frames andselecting regions containing the moving object. In other embodiments, acombination of manual and automatic selection may occur where movingobjects are identified to the user, but the user is allowed to selectwhich region to update.

In order to correctly update or change a selected area after the initialimage is captured, the recaptured selected area may be aligned or mappedonto the initial image. This step may compensate for the user moving thesmartphone (or other portable electronic device) while creating thecomposite image. In techniques as described above, an image frame, asframes are acquired sequentially, will overlap at least partially with apreviously captured image frame. As a result, each image frame may bespatially correlated with the preceding image frames. In this way, thespatial position of each image frame in a continuous sequence of imageframes may be tracked. Such tracking may be performed using a threedimensional point cloud, as described above, or in any other suitableway.

However, once a stop condition occurs, tracking is broken such thatimage frames captured after a resume condition cannot be integrated intoa composite image until a spatial correlation between the new imageframes and the captured composite image is determined. Once determined,though, the tracking may be reestablished and new image frames may beintegrated into the same composite image with image frames capturedbefore the stop condition.

If the user has positioned the camera device such that reestablishingtracking is not possible, this may be indicated to the user through avariety of ways such as a text message, an icon, light indicator, orsound. The indication to the user may also include instructions toreposition the camera onto the scene or take other corrective action.However, if the camera is pointed at a region of the scene that isrepresented in the composite image, relocalization techniques may beused to map the camera position to the positional information of thecaptured image to regain tracking. Such a relocalization techniqueallows the user to move the portable electronic device away from thescene and then return to it to update or change the captured compositeimage. Additionally, relocalization may be used if tracking is inhibitedin other ways, such as by the user moving the smartphone rapidly.

Regardless of how tracking is reestablished, after an area is selectedand the portable electronic device is positioned on the scene, new imageinformation may be integrated with the previously captured compositeimage. Such integration may be performed by adding to the compositeimage or replacing selected portions of the composite image.

Accordingly, once the image frames being captured following the resumecondition are mapped onto the initial image map, the user selectedregion of the initial image may be updated based on the new imageframes. Updating of the composite image, may be performed in anysuitable way. In some embodiments, the updating may occur as the newimage frames are captured, and the updated image may be displayed to theuser. This process of updating the selected region may be repeated untilthe user confirms the update. In some embodiments, the user may be ableto revert the selected area of the image back to a previously capturedimage.

Alternatively, the new image information may be initially recorded as aseparate composite image. When the user provides input indicatingacceptance of the separate composite image, processing may replace theselected portion of the initial composite image with correspondingportions of the separate composite image.

FIG. 29 illustrates an overview of an exemplary dynamic updating process2900 for a composite image. This processing may be implemented in anysuitable processing device. However, in the embodiment illustrated, theprocessing may be performed by a processor of a smartphone used tocapture an image.

Process 2900 may start, for example, when a smartphone operated to scana scene captures an image frame comprising an image of the scene. Theprocessing depicted in FIG. 29 may be initiated by user input or in anyother suitable way. At block 2902, an initial image may be captured. Astop condition may occur after capturing the initial image. Next, atblock 2904, the user selects an area of the composite image to update.This may be done by selecting an area on the user interface of thesmartphone. The selection area is not limited to a particular size orshape and may be designated in any suitable way, such as by touch on atouch sensitive user interface, commands to position a selection box orother graphical user interface element on the screen and/or other userinterface techniques as are known in the art. The selected area may bedepicted on the user interface overlaying the composite image. Once thearea to change is selected, the next frame is recorded at block 2906before proceeding to decision block 2908 that tracks the smartphonemovement and determines whether tracking of the camera movement has beenlost. A resume condition may occur after the area is selected toinitiate recording of the next frame.

If tracking has been lost, then the process, by block 2918, usesrelocalization techniques to find the image position with respect to thepreviously captured composite image. In the embodiment illustrated, therelocalization is based on the camera's positional information, whichmay be determined using sensors in the portable electronic devicecontaining the camera and image matching (such as using the threedimensional point cloud technique as described above) to the compositeimage. The process then takes the next image frame by block 2906 andcontinues with the dynamic updating process 2900.

If the tracking of the camera movement is not lost, then a preview ofthe selected area is updated by block 2910. At this point, the compositeimage may not be finalized and the user may need to accept the updatedpreview of the selected area by block 2912. There are many ways the usercan acknowledge acceptance of the updated selected area and somenon-limiting examples include a button on the camera device and/or userinterface, the user tapping the user interface screen, or a voicerecognition command. If the user has not accepted the preview, then thenext frame is recorded by block 2906 until the preview is accepted. Whenthe user accepts the preview, the selected area of the final compositeimage is updated with the latest captured frame by block 2914. Atdecision block 2916, the user may not be finished recording the imageand choose to select another area to update, thereby returning to block2904. If the user has finished recording, then the final image is savedby block 2920.

FIG. 30A-30F is a time series of images showing an example of a userinteracting with the user interface of a portable electronic device todynamically update a selected area of a composite image. In thisexample, the portable electronic device is a tablet device.

In FIG. 30A, the process of capturing the initial composite image of ascene is initiated. This process continues as the user scans the cameraover the scene. FIG. 30B shows the initial composite image formed as aresult of the scanning.

In FIG. 30C, a region of the initial image is selected to be updated, inthis case the region includes a person in the image. In this example,the area selected for updating is designated by a rectangle overlayingthe displayed composite image. After the person steps out of the scene,further image information may be acquired of that selected region. Theselected area may then be updated with the current image information,thereby removing the person from the composite image, as shown in FIG.30D.

In the illustrated example, the user selects another region in thecomposite image where a person has appeared in the scene in FIG. 30E. Asbefore, this region is updated to now include or add this person intothe composite image in FIG. 30F.

FIG. 31A-31E is another time series of a dynamic updating example whereboth the reality of the scene being captured and what is displayed onthe user interface of the portable electronic device is shown. In thisexample, the illustration on the right represents a composite imagepresented on a user interface of the portable electronic device. Theillustration on the left represents the scene as it exists in reality atthe same time. As can be seen from a comparison of the illustrations onthe left to the illustration on the right in each of FIGS. 31A-31E, thetechniques as described herein enable the composite image to lookdifferent than the scene in reality.

In FIG. 31A, the scene in reality has two people. Initially, the userinterface similarly depicts the scene with two people. As timeprogresses to FIG. 31B, the user interface still displays this initialcomposite image even though, in the scene in reality, one person hasstepped out of the scene.

A third person enters the scene in reality, as depicted in FIG. 31C.Because capture of a composite image is halted, the composite image aspresented on the user interface is not updated. However, the user maydecide to add this person into the composite image by selecting the areaof the image that corresponds to the person's location in the scene. InFIG. 31D, the composite image is updated so this new person is addedinto the image displayed on the user interface. The final image includesall three people, although in reality there are only two people in thescene as shown in FIG. 31E. This exemplifies a solution to the problemof someone being absent from a group image in order to take the image.This dynamic updating technique allows a person in the initial image tostep out of the scene in order for the person initially using theportable electronic device to step into the scene and be included in thecomposite image.

The above-described embodiments of the present invention can beimplemented in any of numerous ways. For example, the embodiments may beimplemented using hardware, software or a combination thereof. Whenimplemented in software, the software code can be executed on anysuitable processor or collection of processors, whether provided in asingle computer or distributed among multiple computers.

Further, it should be appreciated that a computer may be embodied in anyof a number of forms, such as a rack-mounted computer, a desktopcomputer, a laptop computer, or a tablet computer. Additionally, acomputer may be embedded in a device not generally regarded as acomputer but with suitable processing capabilities, including a PersonalDigital Assistant (PDA), a smart phone or any other suitable portable orfixed electronic device.

Also, a computer may have one or more input and output devices. Thesedevices can be used, among other things, to present a user interface.Examples of output devices that can be used to provide a user interfaceinclude printers or display screens for visual presentation of outputand speakers or other sound generating devices for audible presentationof output. Examples of input devices that can be used for a userinterface include keyboards, and pointing devices, such as mice, touchpads, and digitizing tablets. As another example, a computer may receiveinput information through speech recognition or in other audible format.

Such computers may be interconnected by one or more networks in anysuitable form, including as a local area network or a wide area network,such as an enterprise network or the Internet. Such networks may bebased on any suitable technology and may operate according to anysuitable protocol and may include wireless networks, wired networks orfiber optic networks.

Also, the various methods or processes outlined herein may be coded assoftware that is executable on one or more processors that employ anyone of a variety of operating systems or platforms. Additionally, suchsoftware may be written using any of a number of suitable programminglanguages and/or programming or scripting tools, and also may becompiled as executable machine language code or intermediate code thatis executed on a framework or virtual machine.

In this respect, the invention may be embodied as a non-transitorycomputer readable medium (or multiple computer readable media) (e.g., acomputer memory, one or more floppy discs, compact discs (CD), opticaldiscs, digital video disks (DVD), magnetic tapes, flash memories,circuit configurations in Field Programmable Gate Arrays or othersemiconductor devices, or other non-transitory, tangible computerstorage medium) encoded with one or more programs that, when executed onone or more computers or other processors, perform methods thatimplement the various embodiments of the invention discussed above. Thecomputer readable medium or media can be transportable, such that theprogram or programs stored thereon can be loaded onto one or moredifferent computers or other processors to implement various aspects ofthe present invention as discussed above.

The terms “program” or “software” are used herein in a generic sense torefer to any type of computer code or set of computer-executableinstructions that can be employed to program a computer or otherprocessor to implement various aspects of the present invention asdiscussed above. Additionally, it should be appreciated that accordingto one aspect of this embodiment, one or more computer programs thatwhen executed perform methods of the present invention need not resideon a single computer or processor, but may be distributed in a modularfashion amongst a number of different computers or processors toimplement various aspects of the present invention.

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that performs particular tasks or implement particularabstract data types. Typically the functionality of the program modulesmay be combined or distributed as desired in various embodiments.

Also, data structures may be stored in computer-readable media in anysuitable form. For simplicity of illustration, data structures may beshown to have fields that are related through location in the datastructure. Such relationships may likewise be achieved by assigningstorage for the fields with locations in a computer-readable medium thatconveys relationship between the fields. However, any suitable mechanismmay be used to establish a relationship between information in fields ofa data structure, including through the use of pointers, tags or othermechanisms that establish relationship between data elements.

Various aspects of the present invention may be used alone, incombination, or in a variety of arrangements not specifically discussedin the embodiments described in the foregoing and is therefore notlimited in its application to the details and arrangement of componentsset forth in the foregoing description or illustrated in the drawings.For example, aspects described in one embodiment may be combined in anymanner with aspects described in other embodiments.

Also, the invention may be embodied as a method, of which an example hasbeen provided. The acts performed as part of the method may be orderedin any suitable way. Accordingly, embodiments may be constructed inwhich acts are performed in an order different than illustrated, whichmay include performing some acts simultaneously, even though shown assequential acts in illustrative embodiments.

Use of ordinal terms such as “first,” “second,” “third,” etc., in theclaims to modify a claim element does not by itself connote anypriority, precedence, or order of one claim element over another or thetemporal order in which acts of a method are performed, but are usedmerely as labels to distinguish one claim element having a certain namefrom another HI element having a same name (but for use of the ordinalterm) to distinguish the claim elements.

Also, the phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” or “having,” “containing,” “involving,” andvariations thereof herein, is meant to encompass the items listedthereafter and equivalents thereof as well as additional items.

1. A method of forming a composite image from a plurality of imageframes of a scene acquired using a portable electronic device associatedwith a user interface, the method comprising: sequentially processingimage frames of the plurality of image frames by, for a processed imageframe, incorporating the processed image frame into a representation ofthe composite image; receiving user input indicating a region of thecomposite image; and replacing a portion of the representation of thecomposite image based on at least one additional image frame of theplurality of image frames.
 2. The method of claim 1, further comprising:detecting a stop condition; and stopping the sequential processing ofthe plurality of image frames based on the detected stop condition. 3.The method of claim 2, wherein the stop condition comprises a detectedoutput of an inertial sensor on the portable electronic device.
 4. Themethod of claim 1, further comprising: detecting a resume condition; andspatially correlating an image frame of the plurality of image frames tothe representation of the composite image in response to detecting theresume condition.
 5. The method of claim 4, wherein: spatiallycorrelating the image frame to the representation of the composite imagecomprises matching the image frame to an image frame in therepresentation of the composite image.
 6. The method of claim 1,wherein: the plurality of image frames comprises a stream of imageframes; sequentially processing the image frames of the plurality ofimage frames comprises tracking a position of the portable electronicdevice with respect to a spatial coordinate system associated with therepresentation of the composite image; the method further comprises:detecting that the tracking of the position of the portable electronicdevice with respect to the spatial coordinate system was disrupted; andreestablishing tracking of the position of the portable electronicdevice with respect to the spatial coordinate system.
 7. The method ofclaim 6, further comprises: selecting the at least one additional imageframe based on the reestablished tracking.
 8. A method of forming acomposite image from a plurality of image frames of a scene acquiredusing a portable electronic device associated with a user interface, themethod comprising: sequentially processing image frames of a firstportion of the plurality of image frames by, for a processed imageframe, incorporating the processed image frame into a firstrepresentation of the composite image; selecting a region of thecomposite image; determining a spatial correlation between an imageframe in a second portion of the plurality of image frames and therepresentation of the composite image; and replacing a portion of therepresentation of the composite image based on at least one image frameof the second portion of the plurality of image frames.
 9. The method ofclaim 8, wherein: selecting the region of the composite image comprisesdetecting a portion of the composite image representing an object movingthrough the scene.
 10. The method of claim 8, wherein: the methodfurther comprises displaying a representation of the composite image ona user interface; and selecting the region of the composite imagecomprises receiving through the user interface user input designatingthe region of the composite image.
 11. The method of claim 8, whereindetermining a spatial correlation comprises, at least in part, obtainingan output of at least one inertial sensor on the portable electronicdevice at a time when the image frame in the second portion is acquired.12. A portable electronic device, comprising: a camera; a userinterface; and a processor configured to: form a composite image from aplurality of image frames of a scene acquired with the camera associatedwith the user interface; sequentially process the image frames of theplurality of image frames by, for a processed image frame, incorporatingthe processed image frame into a representation of the composite image;receive user input indicating a region of the composite image; andreplace a portion of the representation of the composite image based onat least one additional image frame of the plurality of image frames.13. The processor of claim 12, further configured to: detect a stopcondition; and stop the sequential processing of the plurality of imageframes based on the detected stop condition.
 14. The processor of claim13, wherein the stop condition comprises a detected output of aninertial sensor on the portable electronic device.
 15. The processor ofclaim 12, further configured to: detect a resume conditions; andspatially correlate an image frame of the plurality of image frames tothe representation of the composite image in response to detecting theresume condition.
 16. The processor of claim 15, wherein to spatiallycorrelate the image frame to the representation of the composite imagecomprises matching the image frame to an image frame in therepresentation of the composite image.
 17. The processor of claim 12,wherein: the plurality of image frames comprises a stream of imageframes; sequentially processing the image frames of the plurality ofimage frames comprises tracking a position of the portable electronicdevice with respect to a spatial coordinate system associated with therepresentation of the composite image; the processor further configuredto: detect that the tracking of the position of the portable electronicdevice with respect to the spatial coordinate system was disrupted; andreestablish tracking of the position of the portable electronic devicewith respect to the spatial coordinate system.
 18. The processor ofclaim 17, further configured to: select the at least one additionalimage frame based on the reestablished tracking.
 19. At least onenon-transitory, tangible computer readable storage medium havingcomputer-executable instructions, that when executed by a processor,perform a method of forming a composite image from a plurality of imageframes of a scene acquired using a portable electronic device associatedwith a user interface, the method comprising: sequentially processingimage frames of a first portion of the plurality of image frames by, fora processed image frame, incorporating the processed image frame into afirst representation of the composite image; selecting a region of thecomposite image; determining a spatial correlation between an imageframe in a second portion of the plurality of image frames and therepresentation of the composite image; and replacing a portion of therepresentation of the composite image based on at least one image frameof the second portion of the plurality of image frames.
 20. The methodof claim 19, wherein: selecting the region of the composite imagecomprises detecting a portion of the composite image representing anobject moving through the scene.
 21. The method of claim 19, wherein:the method further comprises displaying a representation of thecomposite image on a user interface; and selecting the region of thecomposite image comprises receiving through the user interface userinput designating the region of the composite image.
 22. The method ofclaim 19, wherein determining a spatial correlation comprises, at leastin part, obtaining an output of at least one inertial sensor on theportable electronic device at a time when the image frame in the secondportion is acquired.